[CentOS] Finding i/o bottleneck

Tue Sep 20 18:55:14 UTC 2011
Nicolas Ross <rossnick-lists at cybercat.ca>

Hi list !

We have a very busy webserver hosted in a clustered environment where the 
document root and data is on a GFS2 partition off a fiber-attached disk 

Now on busy moments, I can see in htop, nmon that there is a fair percentage 
of cpu that is waiting for I/O. In nmon, I can spot that the most busy block 
device correspond to our gfs2 partition where many times, it shows that it's 
100% busy and is read all along.

Now, I want to know what files are being waited for. With lsof I can get a 
listing of open files, but it doesn't gives me if a file is just opended in 
ram or if it's being waited for...

What tools besides lsof, nmon, htop, atop can help me find that info ?

I am under RHEL/CentOS 6.1.