我把nginx日志放到logstash中，并且api信息通过get不幸发送。
所以在logstash中有两个部分存储API信用卡。 这里是例子
message: 10.120.40.105 - - [29/Jul/2015:16:41:09 +0000] "PUT /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=2222222222222222222222222 HTTP/1.1" 200 689 "-" "python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64" "10.120.40.105" 0.180 0.180 request: /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=2222222222222222222222222 我正在通过删除请求
NGUSERNAME [a-zA-Z\.\@\-\+_%]+ NGUSER %{NGUSERNAME} NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time} %{NUMBER:upstream_time} NGINXACCESS %{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:"(?:%{URI:referrer}|-)"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time}
我的input看起来像
grok { match => { "message" => "%{NGINXACCESS}" } patterns_dir => ["/etc/logstash/patterns"] } date { match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ] } geoip { source => "clientip" target => "geoip" database => "/usr/share/GeoIP/GeoLiteCity.dat" add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ] add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ] } mutate { convert => [ "[geoip][coordinates]", "float"] convert => [ "request_time", "float"] convert => [ "upstream_time", "float"] }
是否有任何mutate的方式来取代api_secret =后像“xxxxxxxxxxxx”
谢谢！
这实际上比看起来更难一些，因为mutate的gsub字段实际上并没有做你想要的。 这似乎不像你想像的那么聪明。
我不得不修改你正在使用的模式，捕获request前后的所有内容（ pre_req和post_req分别），但似乎是可能的。
不知道它会如何扩展性能明智，因为有很多过滤在这里进行，但它确实工作。
我用这个configurationtesting了它：
input { stdin {} } filter { grok { match => [ "message" , "(?<pre_req>%{IPORHOST:clientip} (?<ident>[a-zA-Z\.\@\-\+_%]+) (?<auth>[a-zA-Z\.\@\-\+_%]+) \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} )%{URIPATHPARAM:request}(?<post_req> HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:\"(?:%{URI:referrer}|-)\"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time} %{NUMBER:upstream_time})", "message" , "(?<pre_req>%{IPORHOST:clientip} (?<ident>[a-zA-Z\.\@\-\+_%]+) (?<auth>[a-zA-Z\.\@\-\+_%]+) \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} )%{URIPATHPARAM:request}(?<post_req> HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:\"(?:%{URI:referrer}|-)\"|%{QS:referrer}) %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time})" ] break_on_match => true } grok { match => { "request" => "(?<request_path>[^?]*)?(?<request_params>.*)" } } mutate { gsub => [ "request_params" , "[?]", "" ] } kv { field_split => "&" source => "request_params" prefix => "request_params_" } mutate { replace => { "request" => "%{request_path}?api_key=%{request_params_api_key}&api_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX" } replace => { "message" => "%{pre_req}%{request}%{post_req}" } remove_field => [ "request_path", "request_params", "request_params_api_key", "request_params_api_secret", "pre_req", "post_req" ] } } output { stdout { codec => rubydebug } }
而且它似乎已经完成了你想要的
# /opt/logstash/bin/logstash -f config.conf Logstash startup completed 10.120.40.105 - - [29/Jul/2015:16:41:09 +0000] "PUT /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=2222222222222222222222222 HTTP/1.1" 200 689 "-" "python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64" "10.120.40.105" 0.180 0.180 { "message" => "10.120.40.105 - - [29/Jul/2015:16:41:09 +0000] \"PUT /v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX HTTP/1.1\" 200 689 \"-\" \"python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64\" \"10.120.40.105\" 0.180 0.180", "@version" => "1", "@timestamp" => "2015-07-29T19:21:14.678Z", "host" => "elk.example.com", "clientip" => "10.120.40.105", "ident" => "-", "auth" => "-", "timestamp" => "29/Jul/2015:16:41:09 +0000", "verb" => "PUT", "request" => "/v1/resources/scenes/455IrIBcRsa0kkIs6mv9lQ?api_key=11111111111111111&api_secret=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX", "httpversion" => "1.1", "response" => "200", "bytes" => "689", "agent" => "\"python-requests/2.6.0 CPython/2.7.9 Linux/2.6.32-504.30.3.el6.x86_64\"", "xforwardedfor" => "\"10.120.40.105\"", "request_time" => "0.180", "upstream_time" => "0.180" }