如何诊断间歇性缓慢的RAID数据传输速率?

我在Centos 5.5上安装了一个encryption的RAID6。 当我从raid传输一个大的10GB文件到我的电脑时,它会在4-5GB左右,再以9GB暂停。 数据传输从25mb / sec下降到<1mb / sec大约30-60秒。

stream式传输高清video会每隔几分钟就会结束和/或冻结。

传输是通过桑巴。 同样的事情发生在rsync(使用rsync守护进程)上,但传输速率稍高。

谁能告诉我如何去追踪这个问题? 我不知道如何找出瓶颈在哪里。


背景信息:

  • 服务器:CPU是一个Intel(R)Core(TM)2 CPU 6300 @ 1.86GHz
  • 拉姆:4GB
  • 以太网:主板上的千兆位
  • SATA:板载x8,6个端口在一个控制器上,另一个在另一个2上

一个核心的CPU使用率约为45%,同时从RAID中传输数据。

我如何创buildRAID:

Raid堆栈如下:

磁盘 – > RAID – > DM-隐窝 – > LVM – > FS

我有7x2TB磁盘。 每个都有一个'fd'(raid)分区(/ dev / sdb – > / dev / sdh)注意/ dev / sda是​​一个包含OS

这是我如何创build它

创buildRAID设备

mdadm --create /dev/md0 --level=6 --raid-devices=7 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1 

打开encryption数组

 cryptsetup --key-file /mnt/usbstick/img_0768.jpg luksOpen /dev/md0 md0encrypted 

创buildLVM

 pvcreate /dev/mapper/md0encrypted 

创build卷组 – 使用256M,允许约16.7TB。 必须是2的幂

 vgcreate -s 256M VolGroupRaid /dev/mapper/md0encrypted 

创build卷。 (从vgdisplay -v获得了总的PE 37260)

 lvcreate -l 37260 VolGroupRaid --name raidvol 

使用格式

 mkfs.ext3 /dev/VolGroupRaid/raidvol 

安装它

 mount /dev/VolGroupRaid/raidvol /mnt/raid 

系统信息

lspci的

 00:00.0 Host bridge: Intel Corporation 82P965/G965 Memory Controller Hub (rev 02) 00:01.0 PCI bridge: Intel Corporation 82P965/G965 PCI Express Root Port (rev 02) 00:1a.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #4 (rev 02) 00:1a.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #5 (rev 02) 00:1a.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #2 (rev 02) 00:1c.0 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 1 (rev 02) 00:1c.4 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 5 (rev 02) 00:1c.5 PCI bridge: Intel Corporation 82801H (ICH8 Family) PCI Express Port 6 (rev 02) 00:1d.0 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #1 (rev 02) 00:1d.1 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #2 (rev 02) 00:1d.2 USB Controller: Intel Corporation 82801H (ICH8 Family) USB UHCI Controller #3 (rev 02) 00:1d.7 USB Controller: Intel Corporation 82801H (ICH8 Family) USB2 EHCI Controller #1 (rev 02) 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev f2) 00:1f.0 ISA bridge: Intel Corporation 82801HB/HR (ICH8/R) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801HR/HO/HH (ICH8R/DO/DH) 6 port SATA AHCI Controller (rev 02) 00:1f.3 SMBus: Intel Corporation 82801H (ICH8 Family) SMBus Controller (rev 02) 01:00.0 VGA compatible controller: nVidia Corporation G84 [GeForce 8600 GT] (rev a1) 03:00.0 Ethernet controller: Marvell Technology Group Ltd. 88E8053 PCI-E Gigabit Ethernet Controller (rev 22) 04:00.0 SATA controller: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02) 04:00.1 IDE interface: JMicron Technology Corp. JMB362/JMB363 Serial ATA Controller (rev 02) 

dmesg | grep -i raid

 device-mapper: dm-raid45: initialized v0.2594l md: Autodetecting RAID arrays. raid5: automatically using best checksumming function: generic_sse raid5: using function: generic_sse (6816.000 MB/sec) raid6: int64x1 1558 MB/s raid6: int64x2 1937 MB/s raid6: int64x4 1765 MB/s raid6: int64x8 1468 MB/s raid6: sse2x1 3308 MB/s raid6: sse2x2 4863 MB/s raid6: sse2x4 5582 MB/s raid6: using algorithm sse2x4 (5582 MB/s) md: raid6 personality registered for level 6 md: raid5 personality registered for level 5 md: raid4 personality registered for level 4 raid5: device sdh1 operational as raid disk 6 raid5: device sdg1 operational as raid disk 5 raid5: device sdf1 operational as raid disk 4 raid5: device sde1 operational as raid disk 3 raid5: device sdd1 operational as raid disk 2 raid5: device sdc1 operational as raid disk 1 raid5: device sdb1 operational as raid disk 0 raid5: allocated 7412kB for md0 raid5: raid level 6 set md0 active with 7 out of 7 devices, algorithm 2 RAID5 conf printout: 

cat / proc / mdstat

 Personalities : [raid6] [raid5] [raid4] md0 : active raid6 sdh1[6] sdg1[5] sdf1[4] sde1[3] sdd1[2] sdc1[1] sdb1[0] 9767559680 blocks level 6, 64k chunk, algorithm 2 [7/7] [UUUUUUU] unused devices: <none> 

诊断它的方法是删除/replace组件。 您的完整设置是:

  centos(disk-->RAID-->dm-crypt-->LVM-->fs)-->network->mycomputer 

尝试不同的目的地:

  centos(disk-->RAID-->dm-crypt-->LVM-->fs)-->network->anothercomputer 

消除networking:

  centos(disk-->RAID-->dm-crypt-->LVM-->fs)-->centos 

消除所有的堆栈,看看它是否是networking:

  centos(disk(one disk)-->fs)-->network->mycomputer 

如果不只是networking开始添加组件:

  centos(disk-->RAID-->fs)-->network->mycomputer centos(disk-->RAID-->LVM-->fs)-->network->mycomputer 

这些都是我能想到的所有组合。

即使你的最佳传输速率似乎很差,从7驱动器读取一个大的文件RAID 6应该能够很容易地饱和一个GigE链接,所以即使SAMBA的开销,你应该看到比25Meg /秒更好。

encryption开销不应该作为一个普遍的原则,encryption\解密(与AES)100Meg /秒和更多是在一个单一的核心在现代CPU的能力范围内,但是这个消息有关dm-crypt表明它可以成为可能由您的设置引发的瓶颈,这是一个相对较新的,所以我怀疑这仍然是一个问题。

当你说你正在传输到你的计算机 – 那里有什么规格 – 你正在写的驱动器子系统的操作系统,内存和速度有多快,也就是说能够保持25Meg / sec?