VMware ESXi 5性能下降，数百个I / O延迟错误。

我们有一个独立的ESXi5服务器，具有以下硬件规格： – Supermicro X8DTL – IntelXeon®CPU E5506 2.13GHz – 25G内存 – 1TB HD（镜像RAID，本地SATA）

我们有大约17个VM运行，每个〜512MB。 运行Web +数据库服务器。

大约一个月前，我们遇到了服务器崩溃，经过调查，我们在/scratch/log/vobd.log中发现了类似这些错误的错误：

2013-02-21T23:30:14.054Z: [scsiCorrelator] 1657239493834us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T0:L0 performance has improved. I/O latency reduced from 1310595 microseconds to 260642 microseconds. 2013-02-21T23:30:17.888Z: [scsiCorrelator] 1657243328201us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T0:L0 performance has improved. I/O latency reduced from 260642 microseconds to 85292 microseconds. 2013-02-21T23:30:39.275Z: [scsiCorrelator] 1657264714482us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T0:L0 performance has deteriorated. I/O latency increased from average value of 43610 microseconds to 1310310 microseconds. 2013-02-21T23:30:39.275Z: [scsiCorrelator] 1657263440772us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T0:L0 performance has deteriorated. I/O latency increased from average value of 43610 microseconds to 1310310 microseconds. 2013-02-21T23:30:42.796Z: [scsiCorrelator] 1657268235408us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T0:L0 performance has improved. I/O latency reduced from 1310310 microseconds to 257850 microseconds. 2013-02-21T23:30:44.392Z: [scsiCorrelator] 1657269831493us: [vob.scsi.device.io.latency.improved] Device mpx.vmhba2:C0:T0:L0 performance has improved. I/O latency reduced from 257850 microseconds to 86289 microseconds. 2013-02-21T23:32:29.119Z: [scsiCorrelator] 1657374559512us: [vob.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T0:L0 performance has deteriorated. I/O latency increased from average value of 43613 microseconds to 1405607 microseconds. 2013-02-21T23:32:29.120Z: [scsiCorrelator] 1657373285533us: [esx.problem.scsi.device.io.latency.high] Device mpx.vmhba2:C0:T0:L0 performance has deteriorated. I/O latency increased from average value of 43613 microseconds to 1405607 microseconds. 2013-02-21T23:32:35.673Z: [scsiCorrelator] 1657381113191us: [vob.scsi.device.io.latency.improved] Device mp 

在发生崩溃的那一天，我们已经有将近5000个这样的错误，从那以后，我们每天只有2个错误，高达500个错误（虽然没有完整的服务器崩溃）。 在访客虚拟机的正常使用中，我们正在经历读/写慢的困难。 就像查找命令一样简单，会在性能图表中产生较大的峰值。

• 为什么dd第一次显示磁盘速度慢？
• 提高RDPlogin性能
• 是否有令人信服的理由来分开服务器上的BOOT和DATA，以提高性能？
• 疯狂地减缓网站性能
• 为什么我不能预压缩服务器文件？

我们已经取代了HD和RAID控制器。 具有相同的设置和相似数量的虚拟机的服务器没有这些问题。 在第一次崩溃之前（有5k错误的那个），性能还是不错的，但是日志仍然显示同样的错误~30-40次。 在这次崩溃之前的几天，我们为访客虚拟机精简了一个大的（160GB）HD。

以下是（date，popup错误消息的次数，错误之前logging的延迟的平均值（MS）和之后的平均值（MS））。

 2012-10-24 16 976 138,666 2012-10-28 12 1,020 40,421 2012-11-05 16 1,167 273,223 2012-11-06 20 1,226 89,181 2012-11-07 40 1,314 224,957 2012-11-08 48 1,378 165,349 2012-11-09 42 1,441 174,061 2012-11-10 26 1,519 218,381 2012-11-11 8 1,567 112,229 2012-11-12 24 1,593 233,350 2012-11-13 54 1,641 193,695 2012-11-14 80 1,692 222,456 2012-11-15 32 1,738 243,640 2012-11-16 66 1,776 325,366 2012-11-17 30 1,816 176,468 2012-11-18 38 1,850 264,176 2012-11-20 12 1,846 117,589 2012-11-21 34 1,868 252,732 2012-11-22 44 1,895 166,636 2012-11-23 12 1,926 123,632 2012-11-26 4 1,892 98,791 2012-11-27 14 1,899 184,382 2012-11-28 20 1,916 178,908 2012-11-29 10 1,923 134,338 2012-11-30 6 1,923 69,203 2012-12-01 2 1,924 60,052 2012-12-02 4 1,919 122,631 2012-12-03 8 1,898 126,051 2012-12-04 54 1,909 199,758 2012-12-05 462 2,109 394,950 2012-12-06 36 2,228 191,166 2012-12-07 64 2,245 204,348 2012-12-08 32 2,271 294,890 2012-12-10 140 2,290 302,435 2012-12-11 314 2,386 311,973 2012-12-12 150 2,475 261,258 2012-12-13 160 2,532 236,761 2012-12-14 114 2,585 206,043 2012-12-15 84 2,618 211,221 2012-12-16 52 2,640 256,677 2012-12-17 18 2,637 180,975 2012-12-18 62 2,649 228,785 2012-12-19 92 2,669 199,357 2012-12-20 160 2,707 275,119 2012-12-21 124 2,749 245,460 2012-12-22 2 2,763 102,838 2012-12-26 144 2,736 302,383 2012-12-27 140 2,776 292,725 2012-12-28 64 2,813 274,609 2012-12-30 106 2,811 231,112 2012-12-31 148 2,853 295,416 2013-01-01 12 2,881 204,615 2013-01-04 4 2,860 90,300 2013-01-09 246 2,849 279,765 2013-01-10 278 2,909 301,014 2013-01-11 242 2,966 294,417 2013-01-12 92 3,006 308,232 2013-01-14 248 3,036 271,435 2013-01-15 426 3,172 233,094 2013-01-16 388 3,313 276,185 2013-01-17 342 3,423 282,632 2013-01-18 298 3,517 255,919 2013-01-19 232 3,579 287,905 2013-01-20 8 3,611 128,877 2013-01-21 2 3,614 121,942 2013-01-22 142 3,667 265,338 2013-01-23 402 3,738 281,091 2013-01-24 332 3,826 280,295 2013-01-25 178 3,892 270,747 2013-01-26 280 4,018 319,368 2013-01-27 106 4,075 293,760 2013-01-28 610 4,187 213,410 2013-01-29 784 4,700 222,077 2013-01-30 386 5,236 258,133 2013-01-31 4580 8,261 1,681,902 2013-02-01 2 11,211 339,135 2013-02-02 10 38,909 1,200,144 2013-02-04 18 88,573 2,692,687 2013-02-05 190 67,454 2,094,093 2013-02-06 460 58,534 1,858,435 2013-02-07 98 57,683 1,795,912 2013-02-08 62 54,012 1,671,730 2013-02-09 88 52,681 1,711,773 2013-02-10 66 51,016 1,549,408 2013-02-11 84 48,885 1,639,267 2013-02-12 206 48,364 1,829,969 2013-02-13 562 48,651 1,774,433 2013-02-14 170 48,957 1,655,395 2013-02-15 124 47,055 1,550,294 2013-02-16 140 46,099 1,588,326 2013-02-17 110 45,283 1,485,211 2013-02-18 34 43,836 1,356,562 2013-02-19 326 43,608 1,484,757 2013-02-20 224 43,894 1,581,129 2013-02-21 296 43,626 1,568,687 

在这一点上，我们几乎不知所措，我们得到的最好的答案是，由于我们使用SATA驱动器（这可能是一个可怕的想法），我们正在遇到一个很大的瓶颈。 我们正在计划使用SAS驱动器迁移到SAN，但是我们希望确保问题不会跟随我们。

谢谢