debugging高入站stream量

我的一个开发服务器(CentOS 6)突然看到入站networkingstream量大幅上升

路由器流量图(蓝色是入站)

它会减慢抓取的速度,SSHlogin时间超过10秒,打字时出现延迟,网站超时,Nagios因为NRPE检查超时(这是我的Nagios主机)而感到不安,所以似乎突如其来的巨大的纽克交通风暴,但我无法弄清楚它是从哪里来的。 该服务器有一个公共的IP,因此可以直接访问,它运行一个非常严格的IPTables规则集(只允许80,443和几个其他实用程序端口像jenkins)。 我尝试过使用像iftop这样的工具,但是他们没有显示任何与众不同的东西。 不知道这是因为IPTables阻止连接,所以他们没有显示,但因为我不知道这些是外部设备试图连接到我的服务器,或其他。 看起来很奇怪的是,它使SSH变慢,其他服务不响应,但networkingstream量开始大约在同一时间,我开始问题。 我在哪里找出这个stream量来自哪里以及如何阻止它? 我没有直接访问任何路由器,但我可以做任何我想在服务器上。 我查看了/ var / log / messages,还有很多奇怪的关于DNS的消息我从未见过,但是它们看起来并不是错误,只是过度详细的logging(见下文)。

标准有用的东西;

 [sr@ns309372 ~]$ sudo uptime 23:51:41 up 6:30, 3 users, load average: 0.03, 0.12, 0.11 [sr@ns309372 ~]$ sudo free -m total used free shared buffers cached Mem: 3920 2197 1722 0 103 1060 -/+ buffers/cache: 1032 2887 Swap: 1019 0 1019 [sr@ns309372 ~]$ sudo tail -n 30 /var/log/messages Apr 18 23:11:08 ns309372 named[2451]: success resolving 'ftp.halifax.rwth-aachen.de/A' (in 'rwth-aachen.de'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:11:10 ns309372 named[2451]: success resolving 'deneb.dfn.de/AAAA' (in 'dfn.de'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:11:10 ns309372 named[2451]: success resolving 'ns1.leaseweb.nl/AAAA' (in 'leaseweb.nl'?) after disabling EDNS Apr 18 23:11:15 ns309372 named[2451]: success resolving 'ns4.leaseweb.net/AAAA' (in 'leaseweb.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:11:22 ns309372 named[2451]: success resolving 'pkg.jenkins-ci.org/A' (in 'jenkins-ci.org'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/A' (in 'ovh.net'?) after disabling EDNS Apr 18 23:11:30 ns309372 named[2451]: success resolving 'mirror.ovh.net/AAAA' (in 'ovh.net'?) after disabling EDNS Apr 18 23:33:54 ns309372 named[2451]: success resolving 'vs1.nagios.org/A' (in 'nagios.org'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:34:36 ns309372 named[2451]: success resolving 'ns.ripe.net/A' (in 'ripe.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:34:36 ns309372 named[2451]: success resolving 'dns1.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:34:37 ns309372 named[2451]: success resolving 'dns2.ntli.net/A' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:34:38 ns309372 named[2451]: success resolving 'sec1.apnic.net/AAAA' (in 'apnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:34:39 ns309372 named[2451]: success resolving 'sec3.apnic.net/AAAA' (in 'apnic.net'?) after disabling EDNS Apr 18 23:34:40 ns309372 named[2451]: success resolving 'sec3.apnic.net/A' (in 'apnic.net'?) after disabling EDNS Apr 18 23:34:40 ns309372 named[2451]: success resolving 'dns2.ntli.net/AAAA' (in 'ntli.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:35:02 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:35:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:35:56 ns309372 named[2451]: success resolving 'bitbucket.org/A' (in 'bitbucket.org'?) after disabling EDNS Apr 18 23:48:26 ns309372 named[2451]: success resolving '113.155.23.94.in-addr.arpa/PTR' (in '155.23.94.in-addr.arpa'?) after disabling EDNS Apr 18 23:48:29 ns309372 named[2451]: success resolving '8.137.145.217.in-addr.arpa/PTR' (in '217.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:48:29 ns309372 named[2451]: success resolving '10.169.216.196.in-addr.arpa/PTR' (in '169.216.196.in-addr.arpa'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:48:29 ns309372 named[2451]: success resolving 'ns2.lacnic.net/AAAA' (in 'lacnic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/AAAA' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:48:30 ns309372 named[2451]: success resolving 'ns2.dns.br/A' (in 'br'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:48:34 ns309372 named[2451]: success resolving 'ns2.afrinic.net/A' (in 'afrinic.net'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:50:03 ns309372 named[2451]: success resolving 'urlatron.com/A' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:50:04 ns309372 named[2451]: success resolving 'ns2.ecogeek.org/A' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:50:05 ns309372 named[2451]: success resolving 'ns1.ecogeek.org/AAAA' (in 'ecogeek.org'?) after reducing the advertised EDNS UDP packet size to 512 octets Apr 18 23:50:05 ns309372 named[2451]: success resolving 'urlatron.com/AAAA' (in 'urlatron.com'?) after reducing the advertised EDNS UDP packet size to 512 octets [sr@ns309372 ~]$ sudo ifconfig eth0 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51 inet addr:188.165.192.119 Bcast:188.165.192.255 Mask:255.255.255.0 inet6 addr: fe80::227:eff:fe0b:8651/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:456082 errors:0 dropped:91 overruns:0 frame:0 TX packets:821015 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:59793427 (57.0 MiB) TX bytes:1008283171 (961.5 MiB) Interrupt:43 Base address:0xc000 eth0:0 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51 inet addr:94.23.155.32 Bcast:94.23.155.32 Mask:255.255.255.255 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:43 Base address:0xc000 eth0:1 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51 inet addr:94.23.155.113 Bcast:94.23.155.113 Mask:255.255.255.255 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:43 Base address:0xc000 eth0:2 Link encap:Ethernet HWaddr 00:27:0E:0B:86:51 inet addr:178.32.48.78 Bcast:178.32.48.78 Mask:255.255.255.255 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Interrupt:43 Base address:0xc000 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:169675 errors:0 dropped:0 overruns:0 frame:0 TX packets:169675 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:172646550 (164.6 MiB) TX bytes:172646550 (164.6 MiB) [sr@ns309372 ~]$ sudo sar -n DEV 1 3 Linux 2.6.38.2-grsec-xxxx-grs-ipv6-64 (ns309372.ovh.net) 18/04/12 _x86_64_ (2 CPU) 23:57:35 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 23:57:36 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:36 dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:36 eth0 13.00 8.00 1.11 5.08 0.00 0.00 0.00 23:57:36 tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:36 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:36 ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:36 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 23:57:37 lo 10.00 10.00 2.92 2.92 0.00 0.00 0.00 23:57:37 dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:37 eth0 11.00 8.00 0.91 3.47 0.00 0.00 0.00 23:57:37 tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:37 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:37 ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:37 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s 23:57:38 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:38 dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:38 eth0 7.00 9.00 7.54 1.33 0.00 0.00 0.00 23:57:38 tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:38 sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 23:57:38 ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s Average: lo 3.33 3.33 0.97 0.97 0.00 0.00 0.00 Average: dummy0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: eth0 10.33 8.33 3.19 3.30 0.00 0.00 0.00 Average: tunl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: sit0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Average: ip6tnl0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 

我有一些“虚拟”接口来支持多个IP,只有一个物理接口

我会做两件事情:

  1. loggingiptables中的stream量。 您创build一个规则并使用LOG (使用内核日志系统)和ULOG (将日志指向套接字而不是内核系统)目标。 所以你可能会对所有的-A INPUT数据包感兴趣-A INPUT数据包被发送到-j LOG ,可能是--log-prefix "incoming packets "--log-level 6 (它遵循系统日志级别)

  2. 监视您的进程,看看是否有任何收件人是巨大的带宽。 我build议使用nethogs 。 可能是stream量超过了iptables,而不是写入磁盘,而是被一些古怪的进程立即抛弃。