On Tue, Mar 16, 2010 at 10:12:52AM -0400, Ross Walker wrote: > > On two different machines, I've been experiencing disk I/O stalls > > after > > upgrading to the CentOS 5.4 kernel. Both machines have an LSI 1068E > > MPT SAS (mptsas) controller connected to a Chenbro CK13601 36-port SAS > > expander, with one machine having 16 1T WD disks hooked up to it, and > > the other having a mix of about 20 WD/Seagate/Samsung/Hitachi 1T and > > 2T > > disks. > > > > When there's a disk I/O stall, all reads and writes to any disk behind > > the SAS controller/expander just hang for a while (typically for > > almost > > exactly eight seconds), so not just the I/O to one particular disk > > or a > > subset of the disks. The disks on other (on-board SATA) controllers > > still pass I/O requests when the SAS I/O stalls. > > > > I hacked up the attached (dirty) perl script to demonstrate this > > effect > > -- it will read /proc/diskstats in a tight loop, and keep track of > > which request entered the request queue when, and when it completed, > > and > > it will WTF if a request took more than a second. (The same thing can > > probably be done with blktrace, but I was lazy.) New requests get > > submitted, but the pending ones fail to complete for a while, and then > > they all complete at once. > > > > This happens on kernel-2.6.18-164.11.1.el5, while reverting to the > > latest CentOS 5.3 kernel (kernel-2.6.18-128.7.1.el5) makes the issue > > go > > away again, i.e. no more stalls. > > > > It doesn't seem to matter whether the I/O load is high or not -- the > > stalls happen even under almost no load at all. > > > > Before I dig into this further, has anyone experienced anything > > similar? > > A quick google search didn't come up with much. > > I would use iostat -x and see if there is a disk or group of disks > that show abnormal service times and/or utilization. I/O to all 16 disks stalls simultaneously, for 8 seconds at a time, and 'iostat -k 1' shows zero kb/s read and written to each of the disks (sdb - sdq) for the entire interval. > Are there any errors in the logs? Nope. > How are the disks configured? Software raid? Yes, two 8-disk RAID6 sets -- but that doesn't seem relevant. > Is the adapter's firmware at the latest revision? Not sure. I tried upgrading it but the vendor's firmware updater won't let me (see other email for details). > Was .128 kernel running stock drivers? Yes. > Is .164 kernel running stock drivers? Yes. > (maybe weak-updates from .128 kernel?) Nope. > What IO scheduler is this? Default CFQ? Yes.