[CentOS] File system goes read-only once in a while

Fri Aug 1 17:10:44 UTC 2008
Mufit Eribol <hme at onart.com.tr>

Kai Schaetzl wrote:
> You rebooted several times, you did not force a check I think. You have to 
> boot with the rescue CD and then do a thorough fsck on the filesystem. 
> Maybe on each one of the disks separately, don't know. The boot-up check 
> might not be sufficient. There's probably some bad block on the disk that 
> needs to be flagged away. Until this hasn't happened the system will 
> always try to write/read on that sooner or later.
>
> Kai
>   
I used both "#touch /forcefsck" and "#shutdown -rF now" methods. Don't 
they force a check? Actually, now it seems to me they don't. There is no 
sign of starting fsck in messages.

Here is the complete log relevant to disk operations from /var/log/messages

Aug  1 18:29:45 server kernel: rr2310_00:[0 0] Start channel soft reset.
Aug  1 18:29:45 server kernel: rr2310_00:[0 1] Start channel soft reset.
Aug  1 18:29:45 server kernel: rr2310_00:[0 2] Start channel soft reset.
Aug  1 18:29:45 server kernel: rr2310_00:[0 3] Start channel soft reset.
Aug  1 18:29:45 server kernel: rr2310_00:channel [0,0] started successfully
Aug  1 18:29:45 server kernel: rr2310_00:channel [0,1] started successfully
Aug  1 18:29:45 server kernel: rr2310_00:channel [0,2] started successfully
Aug  1 18:29:45 server kernel: rr2310_00:channel [0,3] started successfully
Aug  1 18:29:45 server kernel: scsi8 : rr2310_00
Aug  1 18:29:45 server kernel:   Vendor: HPT       Model: 
DISK_8_0          Rev: 4.00
Aug  1 18:29:45 server kernel:   Type:   
Direct-Access                      ANSI SCSI revision: 00
Aug  1 18:29:45 server kernel: SCSI device sdc: 1562378240 512-byte hdwr 
sectors (799938 MB)
Aug  1 18:29:45 server kernel: sdc: Write Protect is off
Aug  1 18:29:45 server kernel: SCSI device sdc: drive cache: write through
Aug  1 18:29:45 server kernel: SCSI device sdc: 1562378240 512-byte hdwr 
sectors (799938 MB)
Aug  1 18:29:45 server kernel: sdc: Write Protect is off
Aug  1 18:29:45 server kernel: SCSI device sdc: drive cache: write through
Aug  1 18:29:45 server kernel:  sdc: unknown partition table
Aug  1 18:29:45 server kernel: sd 8:0:0:0: Attached scsi disk sdc
Aug  1 18:29:45 server kernel: sd 8:0:0:0: Attached scsi generic sg2 type 0
Aug  1 18:29:45 server kernel: floppy0: no floppy controllers found
Aug  1 18:29:45 server kernel: lp: driver loaded but no devices found
Aug  1 18:29:45 server kernel: ACPI: Power Button (FF) [PWRF]
Aug  1 18:29:45 server kernel: ACPI: Power Button (CM) [PWRB]
Aug  1 18:29:45 server kernel: ibm_acpi: ec object not found
Aug  1 18:29:45 server kernel: md: Autodetecting RAID arrays.
Aug  1 18:29:45 server kernel: md: autorun ...
Aug  1 18:29:45 server kernel: md: ... autorun DONE.
Aug  1 18:29:45 server kernel: device-mapper: multipath: version 1.0.5 
loaded
Aug  1 18:29:45 server kernel: EXT3 FS on md2, internal journal
Aug  1 18:29:45 server kernel: kjournald starting.  Commit interval 5 
seconds
Aug  1 18:29:45 server kernel: EXT3 FS on md1, internal journal
Aug  1 18:29:45 server kernel: EXT3-fs: mounted filesystem with ordered 
data mode.
Aug  1 18:29:45 server kernel: kjournald starting.  Commit interval 5 
seconds
Aug  1 18:29:45 server kernel: EXT3 FS on md0, internal journal
Aug  1 18:29:45 server kernel: EXT3-fs: mounted filesystem with ordered 
data mode.
Aug  1 18:29:45 server kernel: kjournald starting.  Commit interval 5 
seconds
Aug  1 18:29:45 server kernel: EXT3-fs warning (device dm-0): 
ext3_clear_journal_err: Filesystem error recorded from previous mount: 
IO failure
Aug  1 18:29:45 server kernel: EXT3-fs warning (device dm-0): 
ext3_clear_journal_err: Marking fs in need of filesystem check.
Aug  1 18:29:45 server kernel: EXT3-fs warning: mounting fs with errors, 
running e2fsck is recommended
Aug  1 18:29:45 server kernel: EXT3 FS on dm-0, internal journal
Aug  1 18:29:45 server kernel: EXT3-fs: recovery complete.
Aug  1 18:29:45 server kernel: EXT3-fs: mounted filesystem with ordered 
data mode.
Aug  1 18:29:45 server kernel: Adding 6144852k swap on /dev/sda3.  
Priority:1 extents:1 across:6144852k
Aug  1 18:29:45 server kernel: Adding 6144852k swap on /dev/sdb3.  
Priority:1 extents:1 across:6144852k

sdc is the problematic partition (LVM2 on RAID10). I think dm-0 is 
pointing the same device.

If I use a rescue disk loading RAID driver kernel module will not be 
loaded. Can I load it manually? It is getting complicated for me.

Thank you,
Mufit