fioconfiguration来衡量提供商SLA的IOPS

因此,一个提供商给我们提供了500 IOPS / TB作为他们在VMWare&RAID5-SAN环境中的磁盘性能的SLA标准。 这显然是衡量:

  • 平均传输块大小16kB
  • 3:1读取:写入比例
  • multithreadingIO操作
  • 80%的随机IObuild模
  • 读取caching命中20%

我想要做的是确定是否有任何特定的Linux VM正在获得该性能,然后与其他提供者运行相同的基准testing,以便进行比较。

从环顾四周看来, fio似乎是衡量上述的最可configuration的。 我到目前为止的configuration是:

[global] blocksize=16k rwmixread=75 # 3:1 read:write ratio ramp_time=30 runtime=600 time_based buffered=1 # size = free-ram * 80% / 5 # so we get a ~20% cache hit across the 5x processes # this is for an 8GB ram host with 7.3GB free after buffers/cache size=1180m # create a mix to get to 80% random reads # also means we'll be doing at least 5x IO operations in parallel [sla-0] readwrite=randrw:2 [sla-1] readwrite=randrw:2 [sla-2] readwrite=randrw [sla-3] readwrite=randrw [sla-4] readwrite=randrw 

改进build议? 是使用buffered和默认的ioengine最好的方式去?

如果我在8GB内存和470GB分配存储的空闲4x虚拟内核计算机上运行此操作,我希望通过上述(500 * 0.47)获得235 IOPS。 我得到的结果是:

 sla-0: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2 sla-1: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2 sla-2: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2 sla-3: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2 sla-4: (g=0): rw=randrw, bs=16K-16K/16K-16K, ioengine=sync, iodepth=2 Starting 5 processes sla-0: Laying out IO file(s) (1 file(s) / 1180MB) sla-1: Laying out IO file(s) (1 file(s) / 1180MB) sla-2: Laying out IO file(s) (1 file(s) / 1180MB) sla-3: Laying out IO file(s) (1 file(s) / 1180MB) sla-4: Laying out IO file(s) (1 file(s) / 1180MB) Jobs: 5 (f=5): [mmmmm] [100.0% done] [5931K/1966K /s] [362/120 iops] [eta 00m:00s] sla-0: (groupid=0, jobs=1): err= 0: pid=16701 read : io=1086MB, bw=1853KB/s, iops=115, runt=600003msec clat (usec): min=4, max=1771K, avg=8607.53, stdev=22114.44 bw (KB/s) : min= 0, max= 4087, per=24.44%, avg=1914.96, stdev=1130.29 write: io=372416KB, bw=635586B/s, iops=38, runt=600003msec clat (usec): min=6, max=2574, avg=57.38, stdev=79.65 bw (KB/s) : min= 0, max=11119, per=26.07%, avg=679.63, stdev=517.84 cpu : usr=0.08%, sys=0.63%, ctx=64513, majf=0, minf=109 IO depths : 1=107.4%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=69474/23276, short=0/0 lat (usec): 10=10.23%, 20=8.89%, 50=4.15%, 100=11.66%, 250=0.83% lat (usec): 500=1.48%, 750=1.41%, 1000=0.82% lat (msec): 2=0.83%, 4=1.56%, 10=47.07%, 20=5.91%, 50=4.24% lat (msec): 100=0.55%, 250=0.29%, 500=0.06%, 750=0.01%, 1000=0.01% lat (msec): 2000=0.01% sla-1: (groupid=0, jobs=1): err= 0: pid=16702 read : io=963360KB, bw=1605KB/s, iops=100, runt=600180msec clat (usec): min=4, max=2396K, avg=9934.23, stdev=30986.37 bw (KB/s) : min= 0, max= 4657, per=21.64%, avg=1695.89, stdev=1273.00 write: io=326000KB, bw=556206B/s, iops=33, runt=600180msec clat (usec): min=6, max=3882, avg=55.07, stdev=77.92 bw (KB/s) : min= 0, max=10708, per=23.74%, avg=618.92, stdev=559.01 cpu : usr=0.08%, sys=0.53%, ctx=55500, majf=0, minf=129 IO depths : 1=108.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=60210/20375, short=0/0 lat (usec): 10=11.36%, 20=9.63%, 50=3.56%, 100=11.97%, 250=0.81% lat (usec): 500=0.66%, 750=0.50%, 1000=0.37% lat (msec): 2=0.33%, 4=0.74%, 10=49.56%, 20=3.78%, 50=5.48% lat (msec): 100=0.60%, 250=0.43%, 500=0.16%, 750=0.04%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% sla-2: (groupid=0, jobs=1): err= 0: pid=16703 read : io=827584KB, bw=1379KB/s, iops=86, runt=600012msec clat (usec): min=397, max=2396K, avg=11569.59, stdev=31237.03 bw (KB/s) : min= 0, max= 4237, per=18.60%, avg=1457.59, stdev=1113.89 write: io=276192KB, bw=471358B/s, iops=28, runt=600012msec clat (usec): min=8, max=8339, avg=63.95, stdev=121.52 bw (KB/s) : min= 0, max= 8531, per=20.52%, avg=534.85, stdev=478.91 cpu : usr=0.07%, sys=0.54%, ctx=57019, majf=0, minf=89 IO depths : 1=109.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=51724/17262, short=0/0 lat (usec): 10=0.98%, 20=5.38%, 50=3.53%, 100=13.68%, 250=0.92% lat (usec): 500=0.60%, 750=0.39%, 1000=0.22% lat (msec): 2=0.24%, 4=2.26%, 10=59.15%, 20=4.90%, 50=6.28% lat (msec): 100=0.78%, 250=0.48%, 500=0.18%, 750=0.03%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% sla-3: (groupid=0, jobs=1): err= 0: pid=16704 read : io=865920KB, bw=1443KB/s, iops=90, runt=600005msec clat (usec): min=369, max=2396K, avg=11052.97, stdev=32396.85 bw (KB/s) : min= 0, max= 5984, per=19.47%, avg=1525.97, stdev=1164.42 write: io=285568KB, bw=487365B/s, iops=29, runt=600005msec clat (usec): min=7, max=11910, avg=65.72, stdev=154.09 bw (KB/s) : min= 0, max=11064, per=21.38%, avg=557.30, stdev=534.59 cpu : usr=0.07%, sys=0.57%, ctx=59458, majf=0, minf=109 IO depths : 1=109.5%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=54120/17848, short=0/0 lat (usec): 10=0.99%, 20=5.11%, 50=3.58%, 100=13.64%, 250=0.89% lat (usec): 500=0.71%, 750=0.48%, 1000=0.30% lat (msec): 2=0.70%, 4=4.00%, 10=57.63%, 20=5.21%, 50=5.40% lat (msec): 100=0.70%, 250=0.43%, 500=0.16%, 750=0.03%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% sla-4: (groupid=0, jobs=1): err= 0: pid=16705 read : io=934752KB, bw=1558KB/s, iops=97, runt=600007msec clat (usec): min=187, max=2396K, avg=10236.87, stdev=26080.98 bw (KB/s) : min= 0, max=11419, per=20.74%, avg=1625.28, stdev=1338.26 write: io=304528KB, bw=519721B/s, iops=31, runt=600007msec clat (usec): min=7, max=7572, avg=67.29, stdev=117.27 bw (KB/s) : min= 0, max=10772, per=22.06%, avg=575.17, stdev=560.68 cpu : usr=0.08%, sys=0.60%, ctx=63685, majf=0, minf=129 IO depths : 1=108.7%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued r/w: total=58422/19033, short=0/0 lat (usec): 10=0.81%, 20=4.77%, 50=3.62%, 100=13.77%, 250=0.97% lat (usec): 500=1.45%, 750=0.64%, 1000=0.53% lat (msec): 2=1.75%, 4=4.71%, 10=53.48%, 20=6.92%, 50=5.53% lat (msec): 100=0.56%, 250=0.37%, 500=0.08%, 750=0.02%, 1000=0.01% lat (msec): 2000=0.01%, >=2000=0.01% Run status group 0 (all jobs): READ: io=4593MB, aggrb=7836KB/s, minb=1412KB/s, maxb=1897KB/s, mint=600003msec, maxt=600180msec WRITE: io=1528MB, aggrb=2607KB/s, minb=471KB/s, maxb=635KB/s, mint=600003msec, maxt=600180msec Disk stats (read/write): dm-0: ios=298995/596154, merge=0/0, ticks=3107720/433061790, in_queue=436170340, util=99.68%, aggrios=0/0, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% sdb: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=-nan% 

总结每个工作的读写IOPS(为什么fio没有在其摘要中包括这个)?我得到了647,这似乎超过了他们指定的服务级别。 任何显而易见的错误,或者是他们的度量标准对于一些工作负载(特别是我对PostgreSQL的数据仓库工作负载感兴趣)大量偏离。

SQL和数据仓库更像是8:1读写,所有小块,都是随机的。 在任何情况下,除随机读取之外的任何内容都很容易caching,而且不太可能会导致磁盘性能问题。 不知道他们如何做磁盘,真的很难帮助很多,但是当他们指定“RAID5-SAN环境”时,可以考虑询问他们是什么意思。

由于他们将SLA指定为每TB IOPS,所以我猜测他们提供给您的每个卷应该放在单独的RAID-5上,这样在添加卷时可以提供更多的IOPS。 糟糕的性能很容易被糟糕的raid邻居造成:与你相同的raid占用的存储资源相当多。 问题在于有时你的SLA会被超过,但是有时你必须处理高延迟。

首先警告他们你对performance不满意,他们可能会简单地把你移到一个较低的利用率,这可能会解决你所有的问题。 同时询问他们是否有一些可用的raid-10存储空间,并且可能要求提供一个卷,而不是raid 5.如果问题回来了,那么考虑让自己的存储空间或find其他可以提供更好性能的主机。