Logstash Grok匹配模式不工作，即使grok构造函数说没关系

我一直在构build一些grok模式来parsing/ var / log / secure日志文件，一切都工作正常。 我在http://grokconstructor.appspot.com/创buildGrok模式，然后在http://grokdebug.herokuapp.com/上testing它们，并且两个站点都显示完全匹配的模式。 我使用的是使用JAVA openjdk 1.8.0.65-2.b17的所有在CentOS 7.1上运行的logstash 2.1.1，elasticsearch 2.1.1和kibana 4.3.1。

然后我采取了这些模式，并在我的logstash服务器上使用filter来实现它们。 大部分filter都能正常工作，但SECURENETREG和SECURENETBADGE由于某种原因不匹配。 logstash –configtest显示没有问题，否则logstash正常运行，但是当我查看应该匹配那些Kibana模式的条目时，似乎没有任何parsing工作。

这里是我的模式文件/etc/logstash/patterns.d/secure-log.grok：

SECURETIMESTAMP %{MONTH}%{SPACE}%{MONTHDAY} %{TIME} SECUREPROG %{PROG:program}(?:\[%{POSINT:pid}\])? SECUREHOST %{IPORHOST:host} SECUREBASE %{SECURETIMESTAMP:secure_timestamp}%{SPACE}%{SECUREHOST}%{SPACE}%{SECUREPROG}: SECURESU %{SECUREBASE} (runuser: |)%{PROG:pam_program}(?:\[%{POSINT:pid}\])? session %{WORD} for user %{USER:user}( by \(uid=%{NUMBER:su_caller_uid}\)|) SECURESUDORUN %{SECUREBASE}%{SPACE}(%{USER:user} : TTY=%{DATA} ; PWD=%{DATA} ; USER=%{USER:sudo_runas_user} ; COMMAND=%{GREEDYDATA:sudo_command}|\S+:%{SPACE}(TGT verified|error reading keytab %{GREEDYDATA}|authentication %{WORD} for '%{USER:user}'%{GREEDYDATA}|%{GREEDYDATA}user=%{USER:user})) SECURESSHDPUBKEY %{SECUREBASE} (Found matching RSA key: %{GREEDYDATA:rsa_key}|%{WORD} publickey for %{USER:user} from %{IPORHOST:src_ip}( port %{NUMBER:port} %{WORD}( \[preauth\]|: RSA %{GREEDYDATA:rsa_key}|)|)|) SECURESSHDREST %{SECUREBASE} (Did not receive identification string from %{IPORHOST:src_ip}|pam_unix\(sshd:session\): session %{WORD} for user %{USER:user}|Starting session: command for %{USER:user} from %{IPORHOST:src_ip}|Connection from %{IPORHOST:src_ip}|Accepted (password|publickey) for %{USER:user} from %{IPORHOST:src_ip}|Received disconnect from %{IPORHOST:src_ip}|Connection closed by %{IPORHOST:src_ip}|User child is on pid %{NUMBER}|Set %{UNIXPATH} to %{NUMBER}) SECURENETREG %{SECUREBASE} connect from %{IPORHOST:src_ip}%{GREEDYDATA} SECURENETBADGE %{SECUREBASE} (%{WORD:whois_action}|Authentication %{WORD:auth_result}): (reply from %{URI:whois_uri}: Result: %{GREEDYDATA:whois_result}|User: %{USER:user}, %{GREEDYDATA:auth_result_detail}, From: %{IPORHOST:src_ip}, %{GREEDYDATA}, URL: %{URI:netbadge_source_uri}) 

这是我的configuration文件，应用filter，/etc/logstash/conf.d/46-filter-secure-log.conf：

 filter { if [type] == "secure" { grok { patterns_dir => ["/etc/logstash/patterns.d/"] match => { "message" => [ "%{SECURESU}", "%{SECURESUDORUN}", "%{SECURESSHDPUBKEY}", "%{SECURESSHDREST}", "%{SECURENETREG}", "%{SECURENETBADGE}" ] } add_field => { "received_at" => "%{@timestamp}" "received_from" => "%{host}" } } date { match => [ "secure_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] } } } 

最后，这里是一些（识别信息擦除）日志消息，完全匹配在模式testing，但似乎没有得到正确parsinglogstash（虽然我知道filter击中他们，因为我暂时testing通过添加一个新的领域只是为了确保消息正在通过filter运行）：

 Jan 8 09:22:22 netbadge-serv netbadge[3534]: verify_whois: reply from https://whois.domain.edu/whois: Result: 0:-1000:0:0:Error with submitted data Illegal characters in data stream#012 Jan 8 09:22:19 netreg-serv autoreg.pl[13867]: connect from 10.250.100.22 (10.250.100.22) Jan 8 09:22:19 netbadge-serv netbadge[3522]: Authentication success: User: mst3k, Password: Test Test, From: 10.250.28.30, Appid: webmail_login, URL: https://www.mail.domain.edu/switchboard/ Jan 8 09:39:51 netbadge-serv netbadge[11358]: Authentication failure: User: mst3k, Invalid User/Password, From: 10.250.28.31, Appid: Shibboleth Identity Provider, URL: https://shib.domain.edu/idp/Authn/RemoteUser 

我敢肯定，由于盯着这么长时间，我错过了一些简单的东西，所以我希望有人能告诉我这里发生了什么。

谢谢，

短发