HPC群集中的networking连接消失

你好，感谢大家提前！我和我的团队一直困扰着我们用于科学研究的集群的稳定性。我们有很多科学和软件工程经验，但是运行集群的经验不是太多。我会尽量简短。

我们运行一个HPC集群，每个集群有10台机器，每台机器配有4到8个用于科学计算的NVIDIA GeForce 1080 GTX GPU。这些机器本身是Supermicro GPU SuperServers（我们有几个不同的型号）。每个主板都有两个通用网卡，其中只有一个连接到我们的networking。此外，这些机器还具有一个独立的pipe理（IPMI）NIC，这些NIC也连接到同一个networking。注意：所有的NIC都连接到同一个子网。该networking由一个Meraki MX84路由器运行，一个24端口Netgear路由器位于路由器和机器之间。

还有两个特殊的机器; 一个运行MAAS，这是我们用来pipe理群集。另一个是RAID控制器和几个TB的RAID5arrays。所有机器都通过NFS连接到本机。

所有的机器都运行Ubuntu Server 16.04

这些机器位于离我们办公室大约一个小时的托pipe中心。我们有两种连接这些机器的方法：1）将一个VPN插入Meraki提供的networking中，2）通过一个反向隧道将ssh连接到另一台我们在云中运行的机器上。

在正常情况下，我们在GPU机器上运行CPU和GPU密集型作业，这些作业从NFS安装的RAIDarrays中加载必要的数据。

问题是：系统不稳定！在所有事情发生之前，我们无法从这些机器中获得超过几天的运行时间。这是地狱的症状：

大多数机器不能连接到（既不是SSH也不是VPN）。
无法访问的机器也无法通过IPMI访问
有些机器可以连接到，但提供一个非常慢的shell（我的意思是你可以键入命令，但键盘和响应之间存在明显的延迟;感觉非常像networking问题）
我们可以看到的那些机器似乎已经打破了出站互联网连接。具体而言， ping google.com导致DNSparsing问题： unknown host google.com
软重启机器是不够的; 为了恢复function，我们必须通过远程PDU重新上电。

我们的调查显示，我们根本无法进入的机器实际上还活着; 这是一个networking问题，阻止我们访问他们。在post的底部是一个日志，我从一个“死”机器重新启动后拉。您所看到的是正常的DHCP活动，直到凌晨3:00左右才会发生，此时DHCPDISCOVER广播启动失败。当然，在这一点上，ssh隧道（使用autossh运行）开始失败。

我原来的理论是，罪魁祸首是MAAS，因为我们使用的是DHCP服务器，而不是由Meraki路由器提供的。为了testing这个理论，我重新安装了MAAS，这次使用了Meraki的DHCP服务，而不是MAAS。两天之后，系统以标准方式失败，所以我认为我排除了MAAS（至less就DHCP而言）。

我们队中的一些人有一个直觉，认为是NFS。这个理论就像NFS失败，然后一切都变得怪异。我们知道当NFS死亡时，客户机文件系统很难恢复，但不清楚这将如何影响networking。

在这个问题上的任何帮助将是伟大的。就像我说的; 我们没有一个经验丰富的集群经验，所以从哪里开始寻找的指点会很好。更好的办法是针对具体的问题以及如何解决问题。

提前致谢！

日志示例：

 Apr 11 02:02:31 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 252 seconds. Apr 11 02:06:43 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:06:44 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:06:44 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 291 seconds. Apr 11 02:11:35 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:11:35 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:11:35 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 239 seconds. Apr 11 02:15:35 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:15:35 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:15:35 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 275 seconds. Apr 11 02:17:01 cluster9 CRON[7877]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 11 02:20:11 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:20:11 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:20:11 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 250 seconds. Apr 11 02:24:21 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:24:22 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:24:22 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 279 seconds. Apr 11 02:29:01 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:29:01 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:29:01 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 288 seconds. Apr 11 02:33:49 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:33:49 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:33:49 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 281 seconds. Apr 11 02:38:30 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:38:30 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:38:30 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 296 seconds. Apr 11 02:43:26 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:43:26 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:43:26 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 270 seconds. Apr 11 02:47:56 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:47:56 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:47:56 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 260 seconds. Apr 11 02:52:16 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:52:16 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:52:16 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 276 seconds. Apr 11 02:56:52 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 02:56:52 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 02:56:52 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 254 seconds. Apr 11 03:01:06 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 03:01:06 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 03:01:06 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 241 seconds. Apr 11 03:01:30 cluster9 systemd[1]: Started Session 488 of user ubuntu. Apr 11 03:05:07 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 03:05:07 cluster9 dhclient[1558]: DHCPACK of 192.168.128.120 from 192.168.128.101 Apr 11 03:05:07 cluster9 dhclient[1558]: bound to 192.168.128.120 -- renewal in 290 seconds. Apr 11 03:09:57 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8) Apr 11 03:13:51 cluster9 dhclient[1558]: message repeated 18 times: [ DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 192.168.128.101 port 67 (xid=0x5f5d62a8)] Apr 11 03:14:04 cluster9 dhclient[1558]: DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 255.255.255.255 port 67 (xid=0x5f5d62a8) Apr 11 03:15:05 cluster9 dhclient[1558]: message repeated 5 times: [ DHCPREQUEST of 192.168.128.120 on enp3s0f0 to 255.255.255.255 port 67 (xid=0x5f5d62a8)] Apr 11 03:15:08 cluster9 avahi-daemon[1465]: Withdrawing address record for 192.168.128.120 on enp3s0f0. Apr 11 03:15:08 cluster9 avahi-daemon[1465]: Leaving mDNS multicast group on interface enp3s0f0.IPv4 with address 192.168.128.120. Apr 11 03:15:08 cluster9 avahi-daemon[1465]: Interface enp3s0f0.IPv4 no longer relevant for mDNS. Apr 11 03:15:08 cluster9 systemd[1]: Stopping Network Time Synchronization... Apr 11 03:15:08 cluster9 systemd[1]: Stopped Network Time Synchronization. Apr 11 03:15:08 cluster9 systemd[1]: Starting Network Time Synchronization... Apr 11 03:15:08 cluster9 systemd[1]: Started Network Time Synchronization. Apr 11 03:15:08 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 3 (xid=0x3bb49111) Apr 11 03:15:11 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 5 (xid=0x3bb49111) Apr 11 03:15:16 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 11 (xid=0x3bb49111) Apr 11 03:15:27 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 16 (xid=0x3bb49111) Apr 11 03:15:43 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 20 (xid=0x3bb49111) Apr 11 03:16:03 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 16 (xid=0x3bb49111) Apr 11 03:16:19 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 20 (xid=0x3bb49111) Apr 11 03:16:36 cluster9 autossh[13532]: timeout polling to accept read connection Apr 11 03:16:36 cluster9 autossh[13532]: port down, restarting ssh Apr 11 03:16:36 cluster9 autossh[13532]: starting ssh (count 476) Apr 11 03:16:36 cluster9 autossh[13532]: ssh child pid is 8161 Apr 11 03:16:36 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:36 cluster9 autossh[13532]: starting ssh (count 477) Apr 11 03:16:36 cluster9 autossh[13532]: ssh child pid is 8162 Apr 11 03:16:36 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:36 cluster9 autossh[13532]: starting ssh (count 478) Apr 11 03:16:36 cluster9 autossh[13532]: ssh child pid is 8163 Apr 11 03:16:36 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:36 cluster9 autossh[13532]: starting ssh (count 479) Apr 11 03:16:36 cluster9 autossh[13532]: ssh child pid is 8164 Apr 11 03:16:36 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:36 cluster9 autossh[13532]: starting ssh (count 480) Apr 11 03:16:36 cluster9 autossh[13532]: ssh child pid is 8165 Apr 11 03:16:36 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:36 cluster9 autossh[13532]: starting ssh (count 481) Apr 11 03:16:36 cluster9 autossh[13532]: ssh child pid is 8166 Apr 11 03:16:36 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:38 cluster9 autossh[13532]: starting ssh (count 482) Apr 11 03:16:38 cluster9 autossh[13532]: ssh child pid is 8167 Apr 11 03:16:38 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:39 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 19 (xid=0x3bb49111) Apr 11 03:16:46 cluster9 autossh[13532]: starting ssh (count 483) Apr 11 03:16:46 cluster9 autossh[13532]: ssh child pid is 8168 Apr 11 03:16:46 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:16:58 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 19 (xid=0x3bb49111) Apr 11 03:17:01 cluster9 CRON[8170]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly) Apr 11 03:17:04 cluster9 autossh[13532]: starting ssh (count 484) Apr 11 03:17:04 cluster9 autossh[13532]: ssh child pid is 8172 Apr 11 03:17:04 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:17:17 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 11 (xid=0x3bb49111) Apr 11 03:17:28 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 11 (xid=0x3bb49111) Apr 11 03:17:36 cluster9 autossh[13532]: starting ssh (count 485) Apr 11 03:17:36 cluster9 autossh[13532]: ssh child pid is 8173 Apr 11 03:17:36 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:17:39 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 12 (xid=0x3bb49111) Apr 11 03:17:51 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 17 (xid=0x3bb49111) Apr 11 03:18:08 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 11 (xid=0x3bb49111) Apr 11 03:18:19 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 18 (xid=0x3bb49111) Apr 11 03:18:26 cluster9 autossh[13532]: starting ssh (count 486) Apr 11 03:18:26 cluster9 autossh[13532]: ssh child pid is 8174 Apr 11 03:18:26 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:18:37 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 10 (xid=0x3bb49111) Apr 11 03:18:47 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 14 (xid=0x3bb49111) Apr 11 03:19:01 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 21 (xid=0x3bb49111) Apr 11 03:19:22 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 9 (xid=0x3bb49111) Apr 11 03:19:31 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 13 (xid=0x3bb49111) Apr 11 03:19:38 cluster9 autossh[13532]: starting ssh (count 487) Apr 11 03:19:38 cluster9 autossh[13532]: ssh child pid is 8175 Apr 11 03:19:38 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:19:44 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 7 (xid=0x3bb49111) Apr 11 03:19:51 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 8 (xid=0x3bb49111) Apr 11 03:19:59 cluster9 dhclient[1558]: DHCPDISCOVER on enp3s0f0 to 255.255.255.255 port 67 interval 10 (xid=0x3bb49111) Apr 11 03:20:09 cluster9 dhclient[1558]: No DHCPOFFERS received. Apr 11 03:20:09 cluster9 dhclient[1558]: No working leases in persistent database - sleeping. Apr 11 03:21:16 cluster9 autossh[13532]: starting ssh (count 488) Apr 11 03:21:16 cluster9 autossh[13532]: ssh child pid is 8182 Apr 11 03:21:16 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:23:24 cluster9 autossh[13532]: starting ssh (count 489) Apr 11 03:23:24 cluster9 autossh[13532]: ssh child pid is 8183 Apr 11 03:23:24 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:26:06 cluster9 autossh[13532]: starting ssh (count 490) Apr 11 03:26:06 cluster9 autossh[13532]: ssh child pid is 8185 Apr 11 03:26:06 cluster9 autossh[13532]: ssh exited with error status 255; restarting ssh Apr 11 03:26:36 cluster9 autossh[13532]: starting ssh (count 491)