如何提供Elasticsearch我的Snort警报日志?

我昨天从ELK开始,并且很容易地让ELK运行起来。 接下来我想要做的就是把我的Snort警报日志插入它。 我用一个filter和一个绝对令人讨厌的Grok正则expression式来配​​置Logstash(如下所示),使用grokdebug来分割所有的字段来testing它。 然后我打开snort,并且警报日志开始填满,然后重新启动logstash(当然在执行--configtest之后)。 我安装了ES的“头”插件,所以我可以捅一下。 看来,我的snort警报正在映射与我的系统日志映射(如下图所示)。 在ES中,我似乎无法使用logstashconfiguration(ids_proto,src_ip,dst_ip)中定义的任何字段进行search。 为什么是这样? 我需要定义一个映射还是其他东西在这里混乱?

在这里输入图像说明

 input { file { path => "/var/log/snort/alert" type => "snort_tcp" # a type to identify those logs (will need this later) start_position => beginning ignore_older => 0 # Setting ignore_older to 0 disables file age checking so that the tutorial file is processed even though it’s older than a day. sincedb_path => "/dev/null" } } filter { if [type] == "snort_tcp" { grok { add_tag => [ "IDS" ] match => [ "message", "%{MONTHNUM:month}\/%{MONTHDAY:day}-%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}\s+\[\*\*\]\s+\[%{INT:ids_gid}\:%{INT:ids_sid}\:%{INT:ids_rev}\]\s+%{DATA:ids_proto}\s+\[\*\*\]\s+\[Classification:\s+%{DATA:ids_classification}\]\s+\[Priority:\s+%{INT:priority}\]\s+\{%{WORD:ids_proto}\}\s+%{IP:src_ip}\:%{INT:src_port}\s+\-\>\s+%{IP:dst_ip}\:%{INT:dst_port}"] } } geoip { source => "[src_ip]" target => "SrcGeo" } geoip { source => "[dst_ip]" target => "DstGeo" } if [priority] == "1" { mutate { add_field => { "severity" => "High" } } } if [priority] == "2" { mutate { add_field => { "severity" => "Medium" } } } if [priority] == "3" { mutate { add_field => { "severity" => "Low" } } } if [ids_proto] { if [ids_proto] =~ /^GPL/ { mutate { add_tag => [ "Snort-ET-sig" ] add_field => [ "ids_rule_type", "Emerging Threats" ] } } if [ids_proto] =~ /^ET/ { mutate { add_tag => [ "Snort-ET-sig" ] add_field => [ "ids_rule_type", "Emerging Threats" ] } } if "Snort-ET-sig" not in [tags] { mutate { add_tag => [ "Snort-sig" ] add_field => [ "ids_rule_type", "Snort" ] } } } if "Snort-sig" in [tags] { if [ids_gid] == "1" { mutate { add_field => [ "Signature_Info", "http://rootedyour/.com/snortsid?sid=%{ids_sid}" ] } } if [ids_gid] != "1" { mutate { add_field => [ "Signature_Info", "http://rootedyour.com/snortsid?sid=%{ids_gid}-%{ids_sid}" ] } } } if "Snort-ET-sig" in [tags] { mutate { add_field => [ "Signature_Info", "http://doc.emergingthreats.net/bin/view/Main/%{ids_sid}" ] } } } output { elasticsearch { hosts => ["localhost:9200"] manage_template => false index => "snort_tcp-%{+YYYY.MM.dd}" } } 

这里有几件事情:

  • 默认映射logstash创build将所有string字段设置为未分析,这往往是更加友好的下游查看工具。
  • 根本不设置映射,就像使用默认的ElasticSearch dynamic映射一样 ,这种映射不适合Logstash。

为了testing一下,我推荐下面的输出部分:

 output { elasticsearch { hosts => ["localhost:9200"] manage_template => true index => "logstash-%{+YYYY.MM.dd}" } } 

当以这种方式设置时,logstash索引将得到默认的logstash映射,这可能performance得更接近你期望的。 如果是这样的话,你可能需要定义一个映射文件。

 output { elasticsearch { hosts => ["localhost:9200"] manage_template => true index => "snort_tcp-%{+YYYY.MM.dd}" template => "/etc/logstash/template.json" template_name => "snort_tcp" } }