Centos7 – dev sda上的缓冲区I / O错误,逻辑块xxxxxxxxx,asynchronous页写入丢失

我有一个networking服务器,包含HP MSA2040 Storage(10 TB总存储)中的内容。

我不断收到像下面的错误

Jul 31 19:06:24 xxxxxxxx*** kernel: blk_update_request: I/O error, dev sda, sector 7094923416 Jul 31 19:06:24 xxxxxxxx*** kernel: buffer_io_error: 1110 callbacks suppressed Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865171, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865172, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865173, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865174, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865175, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865176, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865177, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865178, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865179, lost async page write Jul 31 19:06:24 xxxxxxxx*** kernel: Buffer I/O error on dev sda, logical block 886865180, lost async page write 

我试图在/ dev / sda上运行xfs_repair这是我的MSA2040存储,这是我得到的报告

 Phase 1 - find and verify superblock... Phase 2 - using internal log - zero log... - scan filesystem freespace and inode maps... - found root inode chunk Phase 3 - for each AG... - scan (but don't clear) agi unlinked lists... - process known inodes and perform inode discovery... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 - process newly discovered inodes... Phase 4 - check for duplicate blocks... - setting up duplicate extent list... - check for inodes claiming duplicate blocks... - agno = 0 - agno = 1 - agno = 2 - agno = 3 - agno = 4 - agno = 5 - agno = 6 - agno = 7 - agno = 8 - agno = 9 No modify flag set, skipping phase 5 Phase 6 - check inode connectivity... - traversing filesystem ... - traversal finished ... - moving disconnected inodes to lost+found ... Phase 7 - verify link counts... 

我甚至试图运行xfs_repaif -L我可以达到我的数据,但一段时间后卡住了。 我也检查了MSA的界面,一切似乎顺利。

有什么方法可以解决这个问题吗?

提前致谢。

编辑*这也是smartctl报告

 === START OF INFORMATION SECTION === Vendor: HP Product: MSA 2040 SAN Revision: G210 User Capacity: 10,239,998,951,424 bytes [10.2 TB] Logical block size: 512 bytes LU is thin provisioned, LBPRZ=1 Rotation Rate: 15000 rpm Logical Unit id: 0x600xxxxxxxxxxxxxxxef5701000000 Serial number: 00c0ff27xxxxxxxxxxxx701000000 Device type: disk Transport protocol: Fibre channel (FCP-2) Local Time is: Mon Jul 31 19:22:30 2017 +03 SMART support is: Available - device has SMART capability. SMART support is: Disabled Temperature Warning: Disabled or Not Supported === START OF READ SMART DATA SECTION === SMART Health Status: OK Elements in grown defect list: 0 Error Counter logging not supported Device does not support Self Test logging 

Edit2 *请求的输出

 [root@xxxxxxxx*** thumbs]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 9.3T 0 disk /msa10tb sdb 8:16 0 1.8T 0 disk ├─sdb1 8:17 0 200M 0 part /boot/efi ├─sdb2 8:18 0 500M 0 part /boot └─sdb3 8:19 0 1.8T 0 part ├─centos-root 253:0 0 50G 0 lvm / ├─centos-swap 253:1 0 7.8G 0 lvm [SWAP] └─centos-home 253:2 0 1.8T 0 lvm /home sr0 11:0 1 1024M 0 rom [root@xxxxxxxx*** thumbs]# pvs PV VG Fmt Attr PSize PFree /dev/sdb3 centos lvm2 a-- 1.82t 64.00m [root@xxxxxxxx*** thumbs]# vgs VG #PV #LV #SN Attr VSize VFree centos 1 3 0 wz--n- 1.82t 64.00m [root@xxxxxxxx*** thumbs]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert home centos -wi-ao---- 1.76t root centos -wi-ao---- 50.00g swap centos -wi-ao---- 7.75g [root@xxxxxxxx*** thumbs]# 

编辑* 3 – 当我检查journalctl,现在我不断得到这些日志;

 Jul 31 19:29:46 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1461891501:1461898801 (repaired) Jul 31 19:29:48 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1461891501:1461898801 (repaired) Jul 31 19:29:50 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1461891501:1461898801 (repaired) Jul 31 19:29:54 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1461891501:1461898801 (repaired) Jul 31 19:30:03 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1461891501:1461898801 (repaired) Jul 31 19:30:25 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1462932481:1462952921 (repaired) Jul 31 19:30:27 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1462932481:1462952921 (repaired) Jul 31 19:30:29 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1462932481:1462952921 (repaired) Jul 31 19:30:30 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:5e37:2374:63273/443 unexpectedly shrunk window 3861953537:3862015916 (repaired) Jul 31 19:30:31 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:5e37:2374:63273/443 unexpectedly shrunk window 3861953537:3862015916 (repaired) Jul 31 19:30:32 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:5e37:2374:63273/443 unexpectedly shrunk window 3861953537:3862015916 (repaired) Jul 31 19:30:33 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:1885:f5b0:30313/443 unexpectedly shrunk window 1462932481:1462952921 (repaired) Jul 31 19:30:34 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:5e37:2374:63273/443 unexpectedly shrunk window 3861953537:3862015916 (repaired) Jul 31 19:30:38 xxxxxxxx*** kernel: Peer 0000:0000:0000:0000:0000:ffff:5e37:2374:63273/443 unexpectedly shrunk window 3861953537:3862015916 (repaired) 

消息为

 Buffer I/O error on dev sda, logical block 886865171, lost async page write 

意味着asynchronous写入(即:脏页写入或缓冲写入)失败。 您在dmesg/var/log/message发现了这些错误,因为失败的asynchronous写入本身不能通知原先提交写入的应用程序。

他们往往是由媒体,一些块不能写的。 这可能是由于:

  • 损坏的磁盘盘片/单元
  • 连接问题(例如:坏电缆,“丢失”的iSCSI目标等)
  • 精简configuration块设备的父池空间被exausted

您直接在文件系统上使用sda ,没有头节点端LVM,所以我们可以排除坏设备映射表作为问题源(至less在头节点上)。

如果您的存储节点的物理属性正常(即:没有错误的磁盘等),我强烈build议查看任何精简configuration的卷及其父池。