> -----Original Message----- > From: Jason Pyeron > Sent: Saturday, February 07, 2015 22:54 > > NOTE: this is happening on Centos 6 x86_64, > 2.6.32-504.3.3.el6.x86_64 not Centos 5 > > Dell PowerEdge 2970, Seagate SATA drive, non-raid. > > I have this server which has been dying randomly, with no logs. Here is a console picture. http://i.imgur.com/ZYHlB82.jpg > > I had a tail -f over ssh for a week, when this just happened. > > Feb 8 00:10:21 thirteen-230 kernel: mptscsih: ioc0: > attempting task abort! (sc=ffff880057a0a080) > Feb 8 00:10:21 thirteen-230 kernel: sd 4:0:0:0: [sda] CDB: > Write(10): 2a 00 1a 17 a1 6f 00 00 01 00 > Feb 8 00:10:51 thirteen-230 kernel: mptscsih: ioc0: WARNING > - Issuing Reset from mptscsih_IssueTaskMgmt!! doorbell=0x24000000 > Feb 8 00:10:51 thirteen-230 kernel: mptbase: ioc0: > Initiating recovery > Feb 8 00:11:13 thirteen-230 kernel: mptscsih: ioc0: task > abort: SUCCESS (rv=2002) (sc=ffff880057a0a080) > Write failed: Connection reset by peer > > After reading https://access.redhat.com/solutions/108273, I > am increasing the logging (shown below) but I am not > confident about this wait and see approach. > > sysctl -w dev.scsi.logging_level=98367 > > I am also going to check smartctl output once I get onsite to > power cycle the system. # smartctl -a /dev/sda smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-504.3.3.el6.x86_64] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Seagate Barracuda (SATA 3Gb/s, 4K Sectors) Device Model: ST1500DM003-9YN16G Serial Number: W24153R0 LU WWN Device Id: 5 000c50 05d03cc1d Firmware Version: CC82 User Capacity: 1,500,301,910,016 bytes [1.50 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Sat Feb 7 23:41:00 2015 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 600) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 194) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 118 099 006 Pre-fail Always - 181943016 3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 17 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 39599363 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 821 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 17 183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 067 062 045 Old_age Always - 33 (Min/Max 30/33) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 16 193 Load_Cycle_Count 0x0032 098 098 000 Old_age Always - 4551 194 Temperature_Celsius 0x0022 033 040 000 Old_age Always - 33 (0 21 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 267112606073648 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2764453802303 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3442873711291 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. > > Other posts I have read, but I can not act on yet: > > * > http://unix.stackexchange.com/questions/34173/mptscsih-ioc0-ta sk-abort-success-rv-2002-causes-30-seconds-freezing > * https://bugzilla.kernel.org/show_bug.cgi?id=18652 > * https://bugzilla.redhat.com/show_bug.cgi?id=483424 > * https://bugzilla.kernel.org/show_bug.cgi?id=42765 > * http://sourceforge.net/p/smartmontools/mailman/message/23849184/ > * http://kb.softescu.ro/category/hardware/dell/ > > -Jason -- -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- - - - Jason Pyeron PD Inc. http://www.pdinc.us - - Principal Consultant 10 West 24th Street #100 - - +1 (443) 269-1555 x333 Baltimore, Maryland 21218 - - - -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- This message is copyright PD Inc, subject to license 20080407P00.
