疑难解答VMWare ESXi 5自发重启

大约3周之前，我将ESXi 5.0服务器迁移到了可丽耐（Colo）服务器上，自那时以来，服务器自发closures并重新启动。 我以前在度假的时候，服务器坐在我的房子里呆了近一个月。 在那段时间里，服务器并没有停机一次。我所知道的唯一区别是：

1. 新的物理位置
2. 安装了Dell PERC5i RAID卡
3. 现在实际上有一些网站，但从交通或处理器的angular度来看，没有什么真正的税收

造成这种情况有点紧急的是，有一次，当ESXi和guest虚拟机恢复运行时，其中一个虚拟机经历了文件系统的恐慌，进入了RO模式。 我重新启动了这个guest，运行fsck，一切恢复正常。 我试图找出是什么原因导致了特定的重新启动，并真正感谢有经验的ESXi用户的眼睛，以发现我的日志中特有的东西。 我没有看到任何看起来像内核崩溃或内存转储的东西。 下面是重新启动事件前后相关日志的摘录…请让我知道，如果我应该包括任何其他人。

vmksummary.log

2012-08-07T17:00:01Z heartbeat: up 2d18h42m11s, 3 VMs; [[3406 vmx 2092436kB] [3453 vmx 2095768kB] [3373 vmx 2300420kB]] [[3531 sfcb-hhrc 2%max] [3432 sfcb-vmware_bas 5%max] [3420 sfcb-pycim 16%max]]
2012-08-07T18:00:01Z heartbeat: up 2d19h42m11s, 3 VMs; [[3406 vmx 2092488kB] [3453 vmx 2095640kB] [3373 vmx 2301544kB]] [[3531 sfcb-hhrc 2%max] [3432 sfcb-vmware_bas 5%max] [3420 sfcb-pycim 16%max]]
2012-08-07T18:58:42Z bootstop: Host has booted
2012-08-07T19:00:01Z heartbeat: up 0d0h2m10s, 3 VMs; [[3405 vmx 464780kB] [3451 vmx 815008kB] [3373 vmx 1086716kB]] [[3501 sfcb-CIMXML-Pro 1%max] [3432 sfcb-vmware_bas 2%max] [3420 sfcb-pycim 5%max]]

• VMware自动重新启动所有主机
• 我如何可靠地从guest虚拟机中确定VMWare Guest的硬件版本？
• 你如何调整Vmware磁盘映像？
• VMware虚拟机pipe理程序版本4.1
• ESXi 5.1 Storage vMotion

syslog.log中

2012-08-04T20:00:01Z crond[2702]: USER root pid 97212 cmd /usr/lib/vmware/vmksummary/log-heartbeat.py
2012-08-04T20:01:01Z crond[2702]: USER root pid 97329 cmd /sbin/auto-backup.sh
2012-08-04T21:00:01Z crond[2702]: USER root pid 99638 cmd /usr/lib/vmware/vmksummary/log-heartbeat.py
2012-08-04T21:01:01Z crond[2702]: USER root pid 99745 cmd /sbin/auto-backup.sh
2012-08-04T22:00:01Z crond[2702]: USER root pid 102014 cmd /usr/lib/vmware/vmksummary/log-heartbeat.py
2012-08-04T22:01:01Z crond[2702]: USER root pid 102081 cmd /sbin/auto-backup.sh
2012-08-04T22:17:54Z jumpstart: dependencies for plugin 'restore-host-cache' not met (missing: vcfs)
2012-08-04T22:17:54Z vmkmicrocode: Warning: Line size is greater than expected size 242
2012-08-04T22:17:54Z vmkmicrocode: File microcode_amd_0x100fa0.bin does not contain a valid microcode update for any of the processors
2012-08-04T22:17:54Z vmkmicrocode: File m4010676860C0001.dat does not contain a valid microcode update for any of the processors
2012-08-04T22:17:54Z vmkmicrocode: File m03106a5.dat does not contain a valid microcode update for any of the processors
2012-08-04T22:17:54Z vmkmicrocode: cpu0 with revision (a07) can use the update in file microcode-1027.dat
2012-08-04T22:17:54Z vmkmicrocode: update number 25 version(1), revision(2571), date(0x9282010), size(2048)
2012-08-04T22:17:54Z vmkmicrocode: cpu1 with revision (a07) can use the update in file microcode-1027.dat
2012-08-04T22:17:54Z vmkmicrocode: update number 25 version(1), revision(2571), date(0x9282010), size(2048)
2012-08-04T22:17:54Z vmkmicrocode: cpu2 with revision (a07) can use the update in file microcode-1027.dat
2012-08-04T22:17:54Z vmkmicrocode: update number 25 version(1), revision(2571), date(0x9282010), size(2048)

vmkernel.log

2012-08-04T02:59:59.509Z cpu4:2655)<6>megasas_hotplug_work[6]: aen event code 0x0027
2012-08-04T15:57:19.630Z cpu5:2655)<6>megasas_hotplug_work[6]: aen event code 0x005e
2012-08-04T16:03:35.776Z cpu4:2649)<6>megasas_hotplug_work[6]: aen event code 0x005e
TSC: 0 cpu0:0)Boot: 167: Parsing boot option module /useropts.gz
TSC: 14715 cpu0:0)Boot: 173: Parsing command line boot options
TSC: 86415 cpu0:0)BootConfig: 38: coresPerPkg = 0
TSC: 90368 cpu0:0)BootConfig: 41: useNUMAInfo = TRUE
TSC: 93878 cpu0:0)BootConfig: 44: numaLatencyLoops = 20
...
PRESUMABLY MORE BOOT STUFF
...
0:00:00:03.667 cpu0:2048)IDT: 991: 0x30 <keyboard> exclusive, flags 0x3
0:00:00:03.667 cpu0:2048)IDT: 991: 0x58 <mouse> exclusive, flags 0x3
0:00:00:03.667 cpu0:2048)IOAPIC: 1335: 0x58 retriggerred
0:00:00:03.667 cpu0:2048)IOAPIC: 1335: 0x30 retriggerred
0:00:00:03.667 cpu0:2048)GlobalTimer: 78: GlobalTimer service not available
0:00:00:03.667 cpu0:2048)Initializing Power Management ...
0:00:00:03.670 cpu0:2048)Power: 2568: No supported CPU power management technology detected
0:00:00:03.671 cpu0:2048)MCE: 616: Fixed 10 MCE bank/CPU-package ownership settings
0:00:00:03.672 cpu0:2048)CpuSched: 11824: Reset scheduler statistics
0:00:00:03.672 cpu0:2048)Init: 892: Vmkernel initialization done. Returning to console.
0:00:00:03.672 cpu0:2048)VMKernel loaded successfully.
2012-08-04T22:17:52.152Z cpu6:2059)ScsiCore: 129: Starting taskMgmt watchdog world 2059
2012-08-04T22:17:52.152Z cpu4:2060)ScsiCore: 129: Starting taskMgmt watchdog world 2060
2012-08-04T22:17:52.152Z cpu5:2141)VSCSI: 2520: Starting reset handler world 2141/1
2012-08-04T22:17:52.152Z cpu3:2177)ScsiCore: 63: Starting taskmgmt handler world 2177/1
2012-08-04T22:17:52.152Z cpu2:2178)ScsiCore: 63: Starting taskmgmt handler world 2178/1
2012-08-04T22:17:52.152Z cpu5:2142)VSCSI: 2709: Starting reset watchdog world 2142

hostd.log

2012-08-04T22:13:54.996Z [FFEA7AC0 info 'Vmomi'] Activation [N5Vmomi10ActivationE:0x33f7abc0] : Invoke done [waitForUpdates] on [vmodl.query.PropertyCollector:ha-property-collector]
2012-08-04T22:13:54.996Z [FFEA7AC0 verbose 'Vmomi'] Arg version:
--> "46"
2012-08-04T22:13:54.996Z [FFEA7AC0 info 'Vmomi'] Throw vmodl.fault.RequestCanceled
2012-08-04T22:13:54.996Z [FFEA7AC0 info 'Vmomi'] Result:
--> (vmodl.fault.RequestCanceled) {
--> dynamicType = <unset>,
--> faultCause = (vmodl.MethodFault) null,
--> msg = "",
--> }
2012-08-04T22:13:54.997Z [34759B90 error 'SoapAdapter.HTTPService'] HTTP Transaction failed on stream TCP(local=127.0.0.1:0, peer=127.0.0.1:58492) with error N7Vmacore15SystemExceptionE(Connection reset by p
2012-08-04T22:14:13.998Z [340C2B90 verbose 'Proxysvc Req01482'] New proxy client TCP(local=66.196.32.10:80, peer=223.4.119.245:43890)
2012-08-04T22:14:44.561Z [348FBB90 verbose 'vm:/vmfs/volumes/4ffd026d-a15e589f-c6e3-003048d37c09/REDACTED/REDACTED.vmx'] Actual VM overhead: 119980032 bytes
2012-08-04T22:14:44.562Z [348FBB90 verbose 'Vmsvc'] RefreshVms updated overhead for 1 VM
2012-08-04T22:15:07.104Z [34718B90 verbose 'Cimsvc'] Ticket issued for CIMOM version 1.0, user root
Section for VMware ESX, pid=2790, version=5.0.0, build=build-623860, option=Release
------ In-memory logs start --------
2012-08-04T22:18:21.746Z [FFC7CAC0 info 'Default'] Supported VMs 87
2012-08-04T22:18:21.746Z [FFC7CAC0 info 'Handle checker'] Setting system limit of 2222
2012-08-04T22:18:21.746Z [FFC7CAC0 info 'Handle checker'] Set system limit to 2222
2012-08-04T22:18:21.746Z [FFC7CAC0 info 'Default'] Setting malloc mmap threshold to 32 k
2012-08-04T22:18:21.746Z [FFC7CAC0 info 'Default'] getrlimit(RLIMIT_NPROC): curr=64 max=128, return code = Success
2012-08-04T22:18:21.746Z [FFC7CAC0 info 'Default'] setrlimit(RLIMIT_NPROC): curr=128 max=128, return code = Success
------ In-memory logs end --------
2012-08-04T22:18:21.747Z [FFC7CAC0 info 'Default'] Initialized channel manager

我排除了：

• 问题与其文件系统R / O的虚拟机 – 我的理解是，单个虚拟机崩溃不能使ESXi
• networkingstream量高峰期的问题 – 该虚拟机上唯一的网站在接近晚上10:30的时候并不是非常繁忙，并且通过访客的Apache日志查看，并且不支持

我猜测：

• 我安装的戴尔RAID卡的问题 – 在安装之前3个星期没事，我将在接下来的几天安装诊断程序，以便我可以监控
• 可能是RAID卡上的吞吐量问题，导致对虚拟机发出的请求响应缓慢，并导致它认为文件系统出了问题，虽然这不应该解释重启，但Linux应该没问题，只是标记FS R / O并继续下去，直到你可以解决问题，如上所述，系统应该不会有负载
• VMWare是否执行需要重启的自动更新？ 我没有在任何guest虚拟机上安装VMWare工具，因此可能导致访客虚拟机重新启动。
• 在科洛的不好的权力 – 移动服务器的那天早上，我不得不让他们重新启动我的机器…我怀疑有人关掉了一个电源或什么东西，因为我得到了一个非常通用的“我们有权力问题”的回应他们。 另外，几个小时前我们遇到了一场大风波，服务器在20分钟的时间内至less重启了3次，没有文件系统损坏，但是对于一个所谓的UPS +发电机支持的数据中心来说，情况并非如此
• 还有什么你能想到的？