[CentOS] disk I/O problems and Solutions

Fri Oct 9 20:35:17 UTC 2009
James A. Peltier <jpeltier at fas.sfu.ca>

On Fri, 9 Oct 2009, Alan McKay wrote:

> Hey folks,
>
> CentOS / PostgreSQL shop over here.
>
> I'm hitting 3 of my favorite lists with this, so here's hoping that
> the BCC trick is the right way to do it :-)
>
> We've just discovered thanks to a new Munin plugin
> http://blogs.amd.co.at/robe/2008/12/graphing-linux-disk-io-statistics-with-munin.html
> that our production DB is completely maxing out in I/O for about a 3
> hour stretch from 6am til 9am
> This is "device utilization" as per the last graph at the above link.

You make no mention of the file system that you are currently using here 
either.  If you look and see I/O wait rather high than your file system 
could also be the culprit.  EXT3/4 journalling is terrible under heavy 
load because the kjournald is single threaded.  We have a user here that 
completely saturates a 4 core 8GB memory machine with only a few clients 
because it can't keep up with the journal writes.

Are you seeing any high I/O waits and lots of kjournald's running?

-- 
James A. Peltier
Systems Analyst (FASNet), VIVARIUM Technical Director
HPC Coordinator
Simon Fraser University - Burnaby Campus
Phone   : 778-782-6573
Fax     : 778-782-3045
E-Mail  : jpeltier at sfu.ca
Website : http://www.fas.sfu.ca | http://vivarium.cs.sfu.ca
           http://blogs.sfu.ca/people/jpeltier
MSN     : subatomic_spam at hotmail.com

Does your OS has a man 8 lart?
http://www.xinu.nl/unix/humour/asr-manpages/lart.html