[CentOS] Journal Aborts in VMware ESX (Filesystem Corruption)

Sun Feb 13 14:09:44 UTC 2011
Adam Tauno Williams <awilliam at whitemice.org>

I have several CentOS5 hosts in a VMware ESX 3.5.0 226117 environment
using iSCSI storage.  Recently we've begun to experience journal aborts
resulting in remounted-read-only filesystems as well as other filesystem
issues - I can unmount a filesystem and force a check with "fsck -f" and
occasionally find errors.

I've found -
<https://bugzilla.redhat.com/show_bug.cgi?id=228108>
<http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=51306>
- which seem related but I believe I am running a kernel that contains
these fixes.

My kernel is 2.6.18-194.32.1.el5 on one of the most effected hosts.

Does anyone else have experience with similar issues or know of the
status of this Bug/KB?

I can install, boot, run, and then at some seemingly random moment -

init_special_inode: bogus i_mode (50632)
init_special_inode: bogus i_mode (137147)
init_special_inode: bogus i_mode (172036)
init_special_inode: bogus i_mode (175720)
init_special_inode: bogus i_mode (72350)
init_special_inode: bogus i_mode (174751)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698169 in dir
#19696695
Aborting journal on device sdb2.
init_special_inode: bogus i_mode (165661)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698131 in dir
#19696695
init_special_inode: bogus i_mode (76763)
init_special_inode: bogus i_mode (3116)
init_special_inode: bogus i_mode (75363)
init_special_inode: bogus i_mode (77034)
init_special_inode: bogus i_mode (132237)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698139 in dir
#19696695
init_special_inode: bogus i_mode (53031)
init_special_inode: bogus i_mode (33361)
init_special_inode: bogus i_mode (77546)
init_special_inode: bogus i_mode (6516)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698143 in dir
#19696695
init_special_inode: bogus i_mode (6442)
init_special_inode: bogus i_mode (72554)
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698142 in dir
#19696695
EXT3-fs error (device sdb2): ext3_lookup: unlinked inode 19698164 in dir
#19696695
init_special_inode: bogus i_mode (73171)
init_special_inode: bogus i_mode (154432)
init_special_inode: bogus i_mode (34302)
init_special_inode: bogus i_mode (131733)
init_special_inode: bogus i_mode (30773)
ext3_abort called.
EXT3-fs error (device sdb2): ext3_journal_start_sb: Detected aborted
journal
Remounting filesystem read-only