[CentOS] Root fs suddenly goes r/o

Thu Jul 12 02:10:30 UTC 2007
redhat at mckerrs.net <redhat at mckerrs.net>

We had exactly this problem on a bunch of 4.4 VMs under ESX 3.0.1. It could happen after 1 day or 6 days, not real pattern except that it was related to the mpt scsi driver timing out afte 5 attempts to write to the target. I found on the VMware technology network forums that it was quite common, and it turns out there was an unofficial kernel patch from redhat to fix the problem. I applied it to my vms and they never suffered again.

I have since upgraded some of those boxes to 4.5 and the related *stock* kernel and the problem appears to have gone away, so it looks like this has been fixed in 4.5 and you dont need to patch.

If you can't upgrade to 4.5 then you can look at the following article for options, NB - when I went look for this stuff there was only a kernel patch, however according to the following article it looks like they have streamlined that to just be a kernel module build;


Hope this helps.



----- Original Message -----
From: "Garrick Staples" <garrick at usc.edu>
To: "CentOS mailing list" <centos at centos.org>
Sent: Thursday, July 12, 2007 9:36:12 AM (GMT+1000) Australia/Brisbane
Subject: Re: [CentOS] Root fs suddenly goes r/o

On Wed, Jul 11, 2007 at 05:18:35PM -0600, Stephen John Smoogen alleged:
> On 7/11/07, Eduardo Grosclaude <eduardo.grosclaude at gmail.com> wrote:
> >Out of the blue, dmesg on my HP Proliant w/ a SCSI disk gives loads of
> >messages like this one:
> >
> > EXT3-fs error (device dm-0) in start_transaction: Journal has aborted
> >
> > Then the root fs goes read-only, so little else can be done on the 
> > machine.
> >LVM locks up. At restart, fs needs a reboot to recover after fsck. The host
> >starts up ok, then I am given some more minutes before the problem
> >reappears. This is stock CentOS 4.4, never have gotten to update it because
> >of this very same problem.
> >
> > System logs say SCSI I/O error, but SMART says no problem has been found,
> >neither does badblocks (run from a rescue CD bootup). SCSI cabling,
> >terminator, etc has been checked.
> >
> > What should I investigate next? Is the disk condemned?
> SMART isnt fool-proof. I have had disks that go 'clunk/scraping
> sounds/spin up' that have gotten SMART seal of approval. My normal
> checklist is the above with replacing the items (in case that isnt
> what you meant by check).
> Replace
> terminator
> scsi cable
> controller
> diskdrive
> though I usually do disk drive then controller.

In my experience the drive is, by far, the most likely to have problems.
Personally I never suspect anything else until I've fully tested the

Garrick Staples, GNU/Linux HPCC SysAdmin
University of Southern California

Please avoid sending me Word or PowerPoint attachments.
See http://www.gnu.org/philosophy/no-word-attachments.html

This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.