[CentOS] Intermittent problem, likely disk IO related - mptscsih: ioc0: attempting task abort!

Sun Feb 8 04:59:38 UTC 2015
Jason Pyeron <jpyeron at pdinc.us>

> -----Original Message-----
> From: Jason Pyeron
> Sent: Saturday, February 07, 2015 22:54
> 
> NOTE: this is happening on Centos 6 x86_64, 
> 2.6.32-504.3.3.el6.x86_64 not Centos 5
> 
> Dell PowerEdge 2970, Seagate SATA drive, non-raid.
> 
> I have this server which has been dying randomly, with no logs.

Here is a console picture.

http://i.imgur.com/ZYHlB82.jpg

> 
> I had a tail -f over ssh for a week, when this just happened.
> 
> Feb  8 00:10:21 thirteen-230 kernel: mptscsih: ioc0: 
> attempting task abort! (sc=ffff880057a0a080)
> Feb  8 00:10:21 thirteen-230 kernel: sd 4:0:0:0: [sda] CDB: 
> Write(10): 2a 00 1a 17 a1 6f 00 00 01 00
> Feb  8 00:10:51 thirteen-230 kernel: mptscsih: ioc0: WARNING 
> - Issuing Reset from mptscsih_IssueTaskMgmt!! doorbell=0x24000000
> Feb  8 00:10:51 thirteen-230 kernel: mptbase: ioc0: 
> Initiating recovery
> Feb  8 00:11:13 thirteen-230 kernel: mptscsih: ioc0: task 
> abort: SUCCESS (rv=2002) (sc=ffff880057a0a080)
> Write failed: Connection reset by peer
> 
> After reading https://access.redhat.com/solutions/108273, I 
> am increasing the logging (shown below) but I am not 
> confident about this wait and see approach.
> 
> sysctl -w dev.scsi.logging_level=98367
> 
> I am also going to check smartctl output once I get onsite to 
> power cycle the system.

# smartctl -a /dev/sda
smartctl 5.43 2012-06-30 r3573 [x86_64-linux-2.6.32-504.3.3.el6.x86_64] (local build)
Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda (SATA 3Gb/s, 4K Sectors)
Device Model:     ST1500DM003-9YN16G
Serial Number:    W24153R0
LU WWN Device Id: 5 000c50 05d03cc1d
Firmware Version: CC82
User Capacity:    1,500,301,910,016 bytes [1.50 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Sat Feb  7 23:41:00 2015 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  600) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 194) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x3085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   118   099   006    Pre-fail  Always       -       181943016
  3 Spin_Up_Time            0x0003   092   092   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       17
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   075   060   030    Pre-fail  Always       -       39599363
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       821
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       17
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   067   062   045    Old_age   Always       -       33 (Min/Max 30/33)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       16
193 Load_Cycle_Count        0x0032   098   098   000    Old_age   Always       -       4551
194 Temperature_Celsius     0x0022   033   040   000    Old_age   Always       -       33 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       267112606073648
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       2764453802303
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       3442873711291

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


> 
> Other posts I have read, but I can not act on yet:
> 
> * 
> http://unix.stackexchange.com/questions/34173/mptscsih-ioc0-ta
sk-abort-success-rv-2002-causes-30-seconds-freezing
> * https://bugzilla.kernel.org/show_bug.cgi?id=18652
> * https://bugzilla.redhat.com/show_bug.cgi?id=483424
> * https://bugzilla.kernel.org/show_bug.cgi?id=42765
> * http://sourceforge.net/p/smartmontools/mailman/message/23849184/
> * http://kb.softescu.ro/category/hardware/dell/
> 
> -Jason


--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.