ElasticSearch延迟索引

我目前有以下设置：

syslog-ng服务器 – > Logstash – > ElasticSearch

系统日志服务器进行负载均衡，并写入到一个SAN位置，Logstash只是将文件拖尾并将它们发送到ES。 我目前正在为networking日志接收约1300个事件/秒的syslog集群。 我遇到的问题是ES日志实际上变得可search时的逐渐延迟。 当我启动群集（4个节点）时，它已经死了。 然后在几分钟后，现在在4天之后，它落后了〜35分钟。 我可以确认日志是在syslog-ng服务器上实时写入的，我也可以确认我的其他4个使用相同概念但不同的Logstash实例的索引保持最新状态。 但是，它们要低得多（约500次/秒）。

看来读取平面文件的Logstash实例不能跟上。 我已经将这些文件分开了一次，并且生成了2个Logstash实例，以帮助它，但我仍然落后。

任何帮助将不胜感激。

–

典型的input是ASA日志，主要是拒绝和VPN连接

Jan 7 00:00:00 firewall1.domain.com Jan 06 2016 23:00:00 firewall1 : %ASA-1-106023: Deny udp src outside:192.168.1.1/22245 dst DMZ_1:10.5.1.1/33434 by access-group "acl_out" [0x0, 0x0] Jan 7 00:00:00 firewall2.domain.com %ASA-1-106023: Deny udp src console_1:10.1.1.2/28134 dst CUSTOMER_094:2.2.2.2/514 by access-group "acl_2569" [0x0, 0x0] 

这是我的Logstashconfiguration。

 input { file { type => "network-syslog" exclude => ["*.gz"] start_position => "end" path => [ "/location1/*.log","/location2/*.log","/location2/*.log"] sincedb_path => "/etc/logstash/.sincedb-network" } } filter { grok { overwrite => [ "message", "host" ] patterns_dir => "/etc/logstash/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns" match => [ "message", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:host} %%{CISCOTAG:ciscotag}: %{GREEDYDATA:message}", "message", "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:host} %{GREEDYDATA:message}" ] } grok { match => [ "message", "%{CISCOFW106001}", "message", "%{CISCOFW106006_106007_106010}", "message", "%{CISCOFW106014}", "message", "%{CISCOFW106015}", "message", "%{CISCOFW106021}", "message", "%{CISCOFW106023}", "message", "%{CISCOFW106100}", "message", "%{CISCOFW110002}", "message", "%{CISCOFW302010}", "message", "%{CISCOFW302013_302014_302015_302016}", "message", "%{CISCOFW302020_302021}", "message", "%{CISCOFW305011}", "message", "%{CISCOFW313001_313004_313008}", "message", "%{CISCOFW313005}", "message", "%{CISCOFW402117}", "message", "%{CISCOFW402119}", "message", "%{CISCOFW419001}", "message", "%{CISCOFW419002}", "message", "%{CISCOFW500004}", "message", "%{CISCOFW602303_602304}", "message", "%{CISCOFW710001_710002_710003_710005_710006}", "message", "%{CISCOFW713172}", "message", "%{CISCOFW733100}", "message", "%{GREEDYDATA}" ] } syslog_pri { } date { "match" => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ] target => "@timestamp" } mutate { remove_field => [ "syslog_facility", "syslog_facility_code", "syslog_severity", "syslog_severity_code"] } } output { elasticsearch { hosts => ["server1","server2","server3"] index => "network-%{+YYYY.MM.dd}" template => "/etc/logstash/logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.2.0-java/lib/logstash/outputs/elasticsearch/elasticsearch-network.json" template_name => "network" } }