在logstash断开的pipe道前的HAProxy

我正在使用loadbalancing设置ELK堆栈。 Haproxy适用于ES和Kibana,但是我遇到了Logstash问题。

这是haproxyconfiguration:

frontend logstash bind 0.0.0.0:5000 mode tcp option tcpka option tcplog log global acl epa hdr_beg(host) -i epa-log acl tgops hdr_beg(host) -i tgops-log use_backend epa_log if epa use_backend tgops_log if tgops backend epa_log mode tcp server elk01 elk01.example.org:5001 check server elk02 elk02.example.org:5001 check server elk03 elk03.example.org:5001 check backend tgops_log mode tcp server elk01 elk01.example.org:5002 check server elk02 elk02.example.org:5002 check server elk03 elk03.example.org:5002 check 

正如你所看到的,即时通讯试图有一个前端服务端口5000和redirect到不同的端口基于源的URL。

如果我直接对服务器做一个netcat它完美的作品

 nc elk01 5001 -vv < /var/log/httpd/error_log 

但是,如果我使用haproxy前端,它会失败,并发生pipe道故障错误。

 nc tgops-log 5000 -vv < /var/log/httpd/error_log Ncat: Version 6.40 ( http://nmap.org/ncat ) libnsock nsi_new2(): nsi_new (IOD #1) libnsock nsock_connect_tcp(): TCP connection requested to 10.129.10.2:5000 (IOD #1) EID 8 libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [10.129.10.2:5000] Ncat: Connected to 10.129.10.2:5000. libnsock nsi_new2(): nsi_new (IOD #2) libnsock nsock_read(): Read request from IOD #1 [10.129.10.2:5000] (timeout: -1ms) EID 18 libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26 libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 26 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 35 [10.129.10.2:5000] libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 42 libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 42 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 51 [10.129.10.2:5000] libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 58 libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 58 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 67 [10.129.10.2:5000] libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 74 libnsock nsock_trace_handler_callback(): Callback: READ EOF for EID 18 [10.129.10.2:5000] libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 74 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 83 [10.129.10.2:5000] libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 90 libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 90 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 99 [10.129.10.2:5000] libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 106 libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 106 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 115 [10.129.10.2:5000] libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 122 libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 122 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE SUCCESS for EID 131 [10.129.10.2:5000] libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 138 libnsock nsock_trace_handler_callback(): Callback: READ SUCCESS for EID 138 [peer unspecified] (8192 bytes) libnsock nsock_trace_handler_callback(): Callback: WRITE ERROR [Broken pipe (32)] for EID 147 [10.129.10.2:5000] Ncat: Broken pipe. 

如果我检查TCP日志,我发现它不会到达后端,抛出一个NOSRV

  logstash logstash/<NOSRV> -1/-1/0 0 SC 15/0/0/0/0 0/0n 10.129.10.82:50724 

我已经尝试了不同的HAProxy选项,如clitcpka ,平衡源,只使用1后端和更多,但没有解决这个问题。

你不能使用mode tcp和匹配头。 mode tcp意味着不要把连接看作是HTTP,而只是传递原始字节。