我的服务器冻结。 我重新启动,手动运行fsck后,服务正常启动。 我正在分析故障发生时的/ var / log / messages,并发现这些条目:
Nov 29 22:24:27 cloud kernel: ata1: exception Emask 0x10 SAct 0x0 SErr 0x4090000 action 0xe frozen Nov 29 22:24:27 cloud kernel: ata1: irq_stat 0x00400040, connection status changed Nov 29 22:24:27 cloud kernel: ata1: SError: { PHYRdyChg 10B8B DevExch } Nov 29 22:24:27 cloud kernel: ata1: hard resetting link Nov 29 22:24:34 cloud kernel: ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300) Nov 29 22:24:34 cloud kernel: ata1.00: configured for UDMA/133 Nov 29 22:24:34 cloud kernel: ata1: EH complete
这可能表明一个硬盘故障,但我做了一个SMART硬盘的长testing,这是输出:
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-573.7.1.el6.x86_64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Seagate Constellation ES (SATA 6Gb/s) Device Model: ST1000NM0011 Serial Number: Z1N2D4JT LU WWN Device Id: 5 000c50 04e008fb2 Firmware Version: PA07 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Mon Nov 30 15:29:22 2015 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 609) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 151) minutes. Conveyance self-test routine recommended polling time: ( 3) minutes. SCT capabilities: (0x10bd) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 069 063 044 Pre-fail Always - 8118850 3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 15 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 084 060 030 Pre-fail Always - 4627909443 9 Power_On_Hours 0x0032 073 073 000 Old_age Always - 24302 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 21 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 075 064 045 Old_age Always - 25 (Min/Max 23/25) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 27 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 29 194 Temperature_Celsius 0x0022 025 040 000 Old_age Always - 25 (0 20 0 0 0) 195 Hardware_ECC_Recovered 0x001a 105 099 000 Old_age Always - 8118850 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 169642618276709 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3860229105 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1373595990 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 24302 - # 2 Short offline Completed without error 00% 24299 - # 3 Short offline Completed without error 00% 24298 - # 4 Short offline Completed without error 00% 4 - # 5 Extended offline Completed without error 00% 4 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
HDD最近已经被取代。 内核版本是2.6.32-573.7.1.el6.x86_64
正如Andrey Sapegin在上面的评论中指出的,这很可能是硬件。
目前我们不能说是HDD还是有线 – 但是很可能是物理上的失败。 他们最近更换了硬盘可以忽略,因为他们没有用新的取代它。
他们说,replaceHDD被认为是好的,因为
他们克隆了一个新的磁盘(使用,但testing),并确认旧磁盘和新磁盘是100%确定,因为在克隆时没有出现错误。
这不是一个说服我的说法。 我在过去的一周内硬盘出现故障,无法写入,但其读取性能非常好。 所以“ 我们从旧硬盘读取所有的数据 ”并没有提到老的健康问题。 “ 我们把所有的数据写入一个新的数据 ”,在我看来,关于replace的健康几乎没有。 如果这是他们的总体testing方法,那么他们对replaceHDD的保证可能被忽略。
谷歌着名的论文清楚地表明 ,只是因为一个驱动器通过smartctl健康检查,你不能知道它是健康的(只有如果它失败了,你肯定知道它不舒服)。
那么这个离开你呢? 你可以肯定你有一个问题,相当肯定的问题是在硬件。 所以做专业的事情:霰弹枪的电缆和驱动器。 硬盘驱动器和电缆很便宜, 根本不值得花时间(或数据风险)来追踪这个错误的来源。 购买新的电缆和新的硬盘驱动器,让您的托pipe公司更换电缆和驱动器。
永远不要再部署没有RAID的服务器。