Elasticsearch性能调优

在单一节点弹性search和logstash一起，我们testing了20mb和200mb文件parsing到Elastic Search的不同types的AWS实例，即Medium，Large和Xlarge。

Logstash conf

input { file { } } filter { mutate { gsub => ["message", "\n", " "] } mutate { gsub => ["message", "\t", " "] } multiline { pattern => "^ " what => "previous" } grok { match => [ "message", "%{TIME:log_time}\|%{WORD:Message_type}\|%{GREEDYDATA:Component}\|%{NUMBER:line_number}\| %{GREEDYDATA:log_message}"] match => [ "path" , "%{GREEDYDATA}/%{GREEDYDATA:loccode}/%{GREEDYDATA:_machine}\:%{DATE:logdate}.log"] break_on_match => false } #To check location is S or L if [loccode] == "S" or [loccode] == "L" { ruby { code => " temp = event['_machine'].split('_') if !temp.nil? || !temp.empty? event['_machine'] = temp[0] end" } } mutate { add_field => ["event_timestamp", "%{@timestamp}" ] replace => [ "log_time", "%{logdate} %{log_time}" ] # Remove the 'logdate' field since we don't need it anymore. lowercase=>["loccode"] remove => "logdate" } # to get all site details (site name, city and co-ordinates) sitelocator{sitename => "loccode" datafile=>"vendor/sitelocator/SiteDetails.csv"} date { locale=>"en" match => [ "log_time", "yyyy-MM-dd HH:mm:ss", "MM-dd-yyyy HH:mm:ss.SSS","ISO8601" ] } } output { elasticsearch{ } } 

环境详细信息：中型实例3.75 RAM 1个内核存储：4 GB SSD 64位networking性能：中等实例运行：Logstash，弹性search

场景：1

• elasticsearch使用批量插入来发布JSON文件
• Elasticsearch：环回接口上的高stream量
• ElasticSearch快照备份滑动到期 – 可能吗？
• 为什么一个节点没有碎片？
• 在kibana仪表板中没有任何显示

 **With default settings** Result : 20mb logfile 23 mins Events Per/second 175 200mb logfile 3 hrs 3 mins Events Per/second 175 Added the following to settings: Java heap size : 2GB bootstrap.mlockall: true indices.fielddata.cache.size: "30%" indices.cache.filter.size: "30%" index.translog.flush_threshold_ops: 50000 indices.memory.index_buffer_size: 50% # Search thread pool threadpool.search.type: fixed threadpool.search.size: 20 threadpool.search.queue_size: 100 **With added settings** Result: 20mb logfile 22 mins Events Per/second 180 200mb logfile 3 hrs 07 mins Events Per/second 180 

情景2

环境详细信息：R3大型15.25 RAM 2核存储：32 GB SSD 64位networking性能：中等实例运行：Logstash，弹性search

 **With default settings** Result : 20mb logfile 7 mins Events Per/second 750 200mb logfile 65 mins Events Per/second 800 Added the following to settings: Java heap size: 7gb other parameters same as above **With added settings** Result: 20mb logfile 7 mins Events Per/second 800 200mb logfile 55 mins Events Per/second 800 

情景3

环境详细信息：R3高内存超大型r3.xlarge 30.5 RAM四核存储：32 GB SSD 64位networking性能：中等实例运行：Logstash，弹性search

 **With default settings** Result: 20mb logfile 7 mins Events Per/second 1200 200mb logfile 34 mins Events Per/second 1200 Added the following to settings: Java heap size: 15gb other parameters same as above **With added settings** Result: 20mb logfile 7 mins Events Per/second 1200 200mb logfile 34 mins Events Per/second 1200 

我想知道

1. 表演的基准是什么？
2. performance是否符合基准，还是低于基准？
3. 为什么即使我增加了弹性searchJVM iam之后，却无法find差异？
4. 我如何监控Logstash并提高其性能？

感谢任何帮助，因为我新的logstash和弹性search。