Hi,
Currently we are running CentOS 4.x on a 2-way Opteron machine. This machine, through a SCSI host adapter (Adaptec), is connected to a 2TB storage unit (an external RAID-5 disk array)
Until our recent unintentional power trip, everything was fine and smooth. We have been experiencing complication accessing the storage ( it could be either intermittent filesystem error, partition could not be mounted in read-write mode, unacceptable writing speed, etc ), especially when we start to 'write' on the storage.
After a few check, we are suspecting either :
1) the storage unit (but the storage control panel did not report any disk/raidset failure) is failing or, 2) the SCSI host adapter is failing, or 3) the filesystem itself is corrupted (we did 'fsck.ext3 -v -f' but it turned out it did not find any errors)
..but we are not sure which one.We did received 'messages' which we never experience previously in our 'dmesg' . (please refer below) Based on the above info and the below 'dmesg' output, we'd appreciate if somebody could share and help us to identify what actually went wrong and how could fix it (if possible)?
TIA.
-Ikmal
Dmesg output : =========================================== scsi4:0:0:0: Attempting to abort cmd 0000010082297a80: 0x28 0x0 0x0 0x0 0x1 0x3f 0x0 0x0 0x8 0x0 scsi4: At time of recovery, card was not paused
>>>>>>>>>>>> Dump Card State Begins
<<<<<<<<<<<<<<<<< scsi4: Dumping Card State at program address 0xc Mode 0x33 Card was paused HS_MAILBOX[0x0] INTCTL[0x80] SEQINTSTAT[0x0] SAVED_MODE[0x11] DFFSTAT[0x33] SCSISIGI[0x0] SCSIPHASE[0x0] SCSIBUS[0x0] LASTPHASE[0x1] SCSISEQ0[0x0] SCSISEQ1[0x12] SEQCTL0[0x0] SEQINTCTL[0x0] SEQ_FLAGS[0xc0] SEQ_FLAGS2[0x0] SSTAT0[0x0] SSTAT1[0x0] SSTAT2[0x0] SSTAT3[0x0] PERRDIAG[0xc0] SIMODE1[0xa4] LQISTAT0[0x0] LQISTAT1[0x0] LQISTAT2[0x0] LQOSTAT0[0x0] LQOSTAT1[0x0] LQOSTAT2[0x0]
SCB Count = 4 CMDS_PENDING = 1 LASTSCB 0xffff CURRSCB 0x2 NEXTSCB 0x0 qinstart = 59 qinfifonext = 59 QINFIFO: WAITING_TID_QUEUES: Pending list: 2 FIFO_USE[0x0] SCB_CONTROL[0x64] SCB_SCSIID[0x7] Total 1 Kernel Free SCB list: 3 1 0 Sequencer Complete DMA-inprog list: Sequencer Complete list: Sequencer DMA-Up and Complete list:
scsi4: FIFO0 Free, LONGJMP == 0x80ff, SCB 0x0 SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x0] DFSTATUS[0x89] SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] SOFFCNT[0x0] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] scsi4: FIFO1 Free, LONGJMP == 0x81d8, SCB 0x3 SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x0] DFSTATUS[0x89] SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] SOFFCNT[0x0] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] LQIN: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 scsi4: LQISTATE = 0x0, LQOSTATE = 0x0, OPTIONMODE = 0x52 scsi4: OS_SPACE_CNT = 0x20 MAXCMDCNT = 0x0 SIMODE0[0xc] CCSCBCTL[0x0] scsi4: REG0 == 0xffff, SINDEX = 0x1e0, DINDEX = 0xe1 scsi4: SCBPTR == 0x3, SCB_NEXT == 0x2, SCB_NEXT2 == 0x2 CDB 28 0 0 80 19 ac STACK: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 <<<<<<<<<<<<<<<<< Dump Card State Ends
>>>>>>>>>>>>
DevQ(0:0:0): 0 waiting (scsi4:A:0:0): Device is disconnected, re-queuing SCB Recovery code sleeping Recovery SCB completes Recovery code awake scsi4: Transmission error detected LQISTAT1[0x0] LASTPHASE[0x1] SCSISIGI[0x0] PERRDIAG[0x1]
>>>>>>>>>>>> Dump Card State Begins
<<<<<<<<<<<<<<<<< scsi4: Dumping Card State at program address 0x26 Mode 0x11 Card was paused HS_MAILBOX[0x0] INTCTL[0x80] SEQINTSTAT[0x0] SAVED_MODE[0x11] DFFSTAT[0x33] SCSISIGI[0x1a] SCSIPHASE[0x1] SCSIBUS[0xff] LASTPHASE[0x1] SCSISEQ0[0x40] SCSISEQ1[0x12] SEQCTL0[0x0] SEQINTCTL[0x0] SEQ_FLAGS[0xc0] SEQ_FLAGS2[0x0] SSTAT0[0x10] SSTAT1[0x11] SSTAT2[0x0] SSTAT3[0x0] PERRDIAG[0x0] SIMODE1[0xac] LQISTAT0[0x0] LQISTAT1[0x0] LQISTAT2[0x0] LQOSTAT0[0x0] LQOSTAT1[0x0] LQOSTAT2[0x0]
SCB Count = 4 CMDS_PENDING = 1 LASTSCB 0xffff CURRSCB 0x2 NEXTSCB 0x0 qinstart = 61 qinfifonext = 61 QINFIFO: WAITING_TID_QUEUES: 0 ( 0x2 ) Pending list: 2 FIFO_USE[0x0] SCB_CONTROL[0x50] SCB_SCSIID[0x7] Total 1 Kernel Free SCB list: 3 1 0 Sequencer Complete DMA-inprog list: Sequencer Complete list: Sequencer DMA-Up and Complete list:
scsi4: FIFO0 Free, LONGJMP == 0x80ff, SCB 0x0 SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x0] DFSTATUS[0x89] SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] SOFFCNT[0x1] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] scsi4: FIFO1 Free, LONGJMP == 0x81d8, SCB 0x3 SEQIMODE[0x3f] SEQINTSRC[0x0] DFCNTRL[0x0] DFSTATUS[0x89] SG_CACHE_SHADOW[0x2] SG_STATE[0x0] DFFSXFRCTL[0x0] SOFFCNT[0x1] MDFFSTAT[0x5] SHADDR = 0x00, SHCNT = 0x0 HADDR = 0x00, HCNT = 0x0 CCSGCTL[0x10] LQIN: 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0 scsi4: LQISTATE = 0x0, LQOSTATE = 0x0, OPTIONMODE = 0x52 scsi4: OS_SPACE_CNT = 0x20 MAXCMDCNT = 0x0 SIMODE0[0xc] CCSCBCTL[0x4] scsi4: REG0 == 0x3, SINDEX = 0x11d, DINDEX = 0xe1 scsi4: SCBPTR == 0x3, SCB_NEXT == 0x2, SCB_NEXT2 == 0x2 CDB 28 0 0 80 19 ac STACK: 0x13 0x0 0x0 0x0 0x0 0x0 0x0 0x0 <<<<<<<<<<<<<<<<< Dump Card State Ends
>>>>>>>>>>>>
DevQ(0:0:0): 0 waiting (scsi4:A:0): 80.000MB/s transfers (40.000MHz DT, 16bit) ===========================================