[CentOS] Intermittent problem, likely disk IO related - mptscsih: ioc0: attempting task abort!
Jason Pyeron
jpyeron at pdinc.us
Sun Feb 8 04:59:38 UTC 2015
> -----Original Message-----
> From: Jason Pyeron
> Sent: Saturday, February 07, 2015 22:54
>
> NOTE: this is happening on Centos 6 x86_64,
> 2.6.32-504.3.3.el6.x86_64 not Centos 5
>
> Dell PowerEdge 2970, Seagate SATA drive, non-raid.
>
> I have this server which has been dying randomly, with no logs.
Here is a console picture.
http://i.imgur.com/ZYHlB82.jpg
>
> I had a tail -f over ssh for a week, when this just happened.
>
> Feb 8 00:10:21 thirteen-230 kernel: mptscsih: ioc0:
> attempting task abort! (sc=ffff880057a0a080)
> Feb 8 00:10:21 thirteen-230 kernel: sd 4:0:0:0: [sda] CDB:
> Write(10): 2a 00 1a 17 a1 6f 00 00 01 00
> Feb 8 00:10:51 thirteen-230 kernel: mptscsih: ioc0: WARNING
> - Issuing Reset from mptscsih_IssueTaskMgmt!! doorbell=0x24000000
> Feb 8 00:10:51 thirteen-230 kernel: mptbase: ioc0:
> Initiating recovery
> Feb 8 00:11:13 thirteen-230 kernel: mptscsih: ioc0: task
> abort: SUCCESS (rv=2002) (sc=ffff880057a0a080)
> Write failed: Connection reset by peer
>
> After reading https://access.redhat.com/solutions/108273, I
> am increasing the logging (shown below) but I am not
> confident about this wait and see approach.
>
> sysctl -w dev.scsi.logging_level=98367
>
> I am also going to check smartctl output once I get onsite to
> power cycle the system.
# smartctl -a /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-504.3.3.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda (SATA 3Gb/s, 4K Sectors)
Device Model: ST1500DM003-9YN16G
Serial Number: W24153R0
LU WWN Device Id: 5 000c50 05d03cc1d
Firmware Version: CC82
User Capacity: 1,500,301,910,016 bytes [1.50 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Sat Feb 7 23:41:00 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 600) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 194) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x3085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 118 099 006 Pre-fail Always - 181943016
3 Spin_Up_Time 0x0003 092 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 17
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 075 060 030 Pre-fail Always - 39599363
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 821
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 17
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 067 062 045 Old_age Always - 33 (Min/Max 30/33)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 16
193 Load_Cycle_Count 0x0032 098 098 000 Old_age Always - 4551
194 Temperature_Celsius 0x0022 033 040 000 Old_age Always - 33 (0 21 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 267112606073648
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2764453802303
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 3442873711291
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
>
> Other posts I have read, but I can not act on yet:
>
> *
> http://unix.stackexchange.com/questions/34173/mptscsih-ioc0-ta
sk-abort-success-rv-2002-causes-30-seconds-freezing
> * https://bugzilla.kernel.org/show_bug.cgi?id=18652
> * https://bugzilla.redhat.com/show_bug.cgi?id=483424
> * https://bugzilla.kernel.org/show_bug.cgi?id=42765
> * http://sourceforge.net/p/smartmontools/mailman/message/23849184/
> * http://kb.softescu.ro/category/hardware/dell/
>
> -Jason
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
- -
- Jason Pyeron PD Inc. http://www.pdinc.us -
- Principal Consultant 10 West 24th Street #100 -
- +1 (443) 269-1555 x333 Baltimore, Maryland 21218 -
- -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.
More information about the CentOS
mailing list