[CentOS] waiting IOs...

Wed Sep 9 18:34:12 UTC 2009
John R Pierce <pierce at hogranch.com>

John Doe wrote:
> Hi,
>
> We have a storage server (HP DL360G5 + MSA20 (12 disks in RAID 6) on a SmartArray6400).
> 10 directories are exported through nfs to 10 clients (rsize=32768,wsize=32768,soft,intr,nosuid,proto=udp,vers=3).
> The server is apparently not doing much but... we have very high waiting IOs.
>
> dstat show very little activity, but high 'wai'...
>   

as others have said, iostat -x N  for an N like 5 (5 seconds).   ignore 
the first sample as its the average since boot, instead, look at the 
ongoing 5 second interval samples, and look for high await, svctm, and 
%util, as well as the rrqm/s and wrqm/s numbers, rather than sec/s, as 
sequential access is likely the least of your problems.    raid6 does 
pretty poorly on heavy random write workloads

Also, IBM has a neat freeware system analysis tool called NMON 
(originally for AIX, ported to Linux)
http://www.ibm.com/developerworks/aix/library/au-analyze_aix/     works 
sorta like a souped up 'top' but has per file system IO stats and stuff 
too.   it can also accumulate stats over a long period into a CSV file, 
and they have an excel spreadsheet that loads said CSV file and cranks 
out a lot of fairly useful graphs.    dunno if the excel spreadsheet 
works in OOcalc or not.