I got a I/O error on one of my disks that caused the journal to abort (error below) and be mounted in read only mode. I rebooted and the file system check ran automatically and resolved the error. I did some searching around and couldn't find a likely cause (the disk is a virtual disk, shared from a SAN for VMWare ESXi5.0).
An hour after bringing the system back up, we got another one. Same sort of error, different inode. Went to run level 1, and ran fsck manually (running now).
I don't see any obvious errors in the vmware-log file for the VM. Or on the SAN. All the other VMs are running fine and I don't see any issues there.
Is there a good process to follow when trying to debug an I/O error like this? There are so many variables in this case. Wasn't sure if there were any ext3 utilities that I don't know about but should be using
Error: Oct 18 02:52:38 shiva clamd[3978]: SelfCheck: Database status OK. Oct 18 03:00:03 shiva kernel: end_request: I/O error, dev sdb, sector 51083840 Oct 18 03:00:03 shiva kernel: Aborting journal on device sdb10. Oct 18 03:00:03 shiva kernel: ext3_abort called. Oct 18 03:00:03 shiva kernel: EXT3-fs error (device sdb10): ext3_journal_start_sb: Detected aborted journal Oct 18 03:00:03 shiva kernel: Remounting filesystem read-only Oct 18 03:00:03 shiva kernel: EXT3-fs error (device sdb10) in ext3_reserve_inode_write: Journal has aborted Oct 18 03:00:03 shiva kernel: EXT3-fs error (device sdb10) in ext3_reserve_inode_write: Journal has aborted Oct 18 03:02:38 shiva clamd[3978]: SelfCheck: Database status OK.