iSCSI到NetApp FAS：写入负载的高延迟

问题

写入NetApp FAS目标时，Linux iSCSI启动器的服务时间很长，暴露了一堆LUN。 在哪里寻找原因以及如何解决这个问题？

再生产

我正在使用iostats包中的iostats和sa来计算“await” – 一个特定请求平均等待的时间：

 dd if=/dev/urandom of=test bs=8K count=1000000 & iostat -xdm 5 sdy dm-26 [...] Device: rrqm/s wrqm/sr/sw/s rMB/s wMB/s avgrq-sz avgqu-sz await svctm %util sdy 0.00 1115.60 0.10 47.50 0.00 7.94 341.68 1.63 34.05 2.00 9.52 dm-26 0.00 0.00 0.10 2032.90 0.00 7.94 8.00 328.10 161.39 0.05 9.52 

在这种情况下，“等待”的iostats测量的值低一个数量级。 CloudShark提供了上述样本的时间段内传输的10秒的networkingstream量。 dm-26是托pipe涉及sdy “物理”设备的文件系统（单磁盘NSS卷）的设备映射器设备。

环境

发起者和目标被放置在相同的子网中。 启动器主机的IP为192.168.20.72，目标为192.168.20.33，stream量切换为1GbE，启用巨帧（通过networking跟踪确认使用），启用iSCSI立即数据（并且正在使用每提到的跟踪），摘要不启用。

• 在ubuntu中无人值守升级后自动构build内核模块
• 最大传输速率iSCSI绑定与多path
• 如何通过股票CMD添加一个iscsi目标到Windows发起者？
• iSCSI启动器即使在ISCSI目标服务器上删除后也显示ISCSI目标
• EMC NS-480 Celerra – validation哪些iSCSI启动器当前连接到LUN的方法？

iSCSI会话信息：

 iscsiadm -m session -P 3 iSCSI Transport Class version 2.0-870 version 2.0-873.suse Target: iqn.1992-08.com.netapp:sn.151745715 Current Portal: 192.168.20.33:3260,2003 Persistent Portal: 192.168.20.33:3260,2003 ********** Interface: ********** Iface Name: default Iface Transport: tcp Iface Initiatorname: iqn.2015-06.de.example.dom:01:gw-cl-07 Iface IPaddress: 192.168.20.72 Iface HWaddress: <empty> Iface Netdev: <empty> SID: 1 iSCSI Connection State: LOGGED IN iSCSI Session State: LOGGED_IN Internal iscsid Session State: NO CHANGE ********* Timeouts: ********* Recovery Timeout: 120 Target Reset Timeout: 30 LUN Reset Timeout: 30 Abort Timeout: 15 ***** CHAP: ***** username: <empty> password: ******** username_in: <empty> password_in: ******** ************************ Negotiated iSCSI params: ************************ HeaderDigest: None DataDigest: None MaxRecvDataSegmentLength: 262144 MaxXmitDataSegmentLength: 65536 FirstBurstLength: 65536 MaxBurstLength: 65536 ImmediateData: Yes InitialR2T: No MaxOutstandingR2T: 1 ************************ Attached SCSI devices: ************************ Host Number: 3 State: running scsi3 Channel 00 Id 0 Lun: 0 Attached scsi disk sdb State: running scsi3 Channel 00 Id 0 Lun: 1 Attached scsi disk sdc State: running scsi3 Channel 00 Id 0 Lun: 10 Attached scsi disk sdl State: running scsi3 Channel 00 Id 0 Lun: 11 Attached scsi disk sdm State: running scsi3 Channel 00 Id 0 Lun: 12 Attached scsi disk sdn State: running scsi3 Channel 00 Id 0 Lun: 13 Attached scsi disk sdo State: running scsi3 Channel 00 Id 0 Lun: 14 Attached scsi disk sdp State: running scsi3 Channel 00 Id 0 Lun: 15 Attached scsi disk sdq State: running scsi3 Channel 00 Id 0 Lun: 16 Attached scsi disk sdr State: running scsi3 Channel 00 Id 0 Lun: 17 Attached scsi disk sds State: running scsi3 Channel 00 Id 0 Lun: 18 Attached scsi disk sdt State: running scsi3 Channel 00 Id 0 Lun: 19 Attached scsi disk sdu State: running scsi3 Channel 00 Id 0 Lun: 2 Attached scsi disk sdd State: running scsi3 Channel 00 Id 0 Lun: 20 Attached scsi disk sdv State: running scsi3 Channel 00 Id 0 Lun: 21 Attached scsi disk sdw State: running scsi3 Channel 00 Id 0 Lun: 22 Attached scsi disk sdx State: running scsi3 Channel 00 Id 0 Lun: 23 Attached scsi disk sdy State: running scsi3 Channel 00 Id 0 Lun: 24 Attached scsi disk sdz State: running scsi3 Channel 00 Id 0 Lun: 25 Attached scsi disk sdaa State: running scsi3 Channel 00 Id 0 Lun: 26 Attached scsi disk sdab State: running scsi3 Channel 00 Id 0 Lun: 27 Attached scsi disk sdac State: running scsi3 Channel 00 Id 0 Lun: 28 Attached scsi disk sdad State: running scsi3 Channel 00 Id 0 Lun: 3 Attached scsi disk sde State: running scsi3 Channel 00 Id 0 Lun: 4 Attached scsi disk sdf State: running scsi3 Channel 00 Id 0 Lun: 5 Attached scsi disk sdg State: running scsi3 Channel 00 Id 0 Lun: 6 Attached scsi disk sdh State: running scsi3 Channel 00 Id 0 Lun: 7 Attached scsi disk sdi State: running scsi3 Channel 00 Id 0 Lun: 8 Attached scsi disk sdj State: running scsi3 Channel 00 Id 0 Lun: 9 Attached scsi disk sdk State: running 

其他意见

由于某种原因，映射到“物理”LUN的dm设备在请求队列中合并写入请求时的等待时间显着增加。 但是我的问题实际上是关于底层设备的等待 – NetApp FAS应该简单地将所有写入请求放入其NVRAM中，即使是同步写入也要立即确认，所以我永远不应该看到等待时间高于5ms，只要networking链接不是饱和的（不是这样的），而且NVRAM不是反压的（这不是–FAS目前不处理任何其他负载）。

读取操作的“等待”时间相当低，即使是随机读取也是如此。 从iozone运行随机读取/写入testing的间隔（启用了O_DSYNC，8K块大小的单线程自动typestesting运行）开始，vizualized 10秒的sysstat数据显示出以下效果： 图的前半部分是随机读取，运行时间约2-4 kIOPS，延迟时间约3ms。 在下半年，工作负载将转换为随机写入，等待上升到> 10ms，IOPS下降到100（负载是延迟绑定和单线程的，所以IOPS与延迟成反比）

出于某种原因，在分析上面的networkingstream量捕获时，Wireshark的SCSI服务响应时间统计function无法识别大部分的write调用，只发现19个请求，并报告平均服务响应时间为3ms，平均SRT值类似于等待34毫秒。

使用的Linux是SuSE SLES 11 SP3，内核3.0.101-0.47.55-默认。