[CentOS] disk I/O problems and Solutions

[Ray Van Dolson]

On Fri, Oct 09, 2009 at 01:34:17PM -0400, Alan McKay wrote:
> On Fri, Oct 9, 2009 at 1:28 PM, Ray Van Dolson <rayvd at bludgeon.org> wrote:
> > Really hard to say what's going on.  Does your DB need optimization?
> > Do the applications hitting it?  Maybe some indexing?  Maybe some more
> > RAM on the machine would help?  What exactly is the workload like --
> > especially during the time when you're peaked out?
> 
> Yes, these are all things we are just starting to look at - so no
> clear answers yet
> 
> > Is the system swapping?  If so, you either need more memory or need to
> > track down a memory leak.... 'free' and 'sar' can both help you see
> > what swap usage is like.
> 
> That's one thing it does not seem to be doing, fortunatley.
> 
> > Writes are always slower on any parity based RAID setup, so I imagine
> > you'd get superior performance on RAID10, especially if you're write
> > heavy.
> 
> We are generally read heavy but at certain times of day including 6 to
> 9am we have huge batches of writes.
> 
> > But to begin with, it'd be interesting to know exactly what this server
> > is doing.  Does it makes sense that the disks are being brought to
> > their knees with the given workload?
> 
> Oh yes!   We are a financial services provider and the Db contains all
> of the worlds stock prices for about the last 10 years :-)  And every
> day new batches of prices come in (at different times depending on
> markets).   This system is just a PostgreSQL DB.   Nothing else.

I guess disk prices are cheap enough these days that it might be enough
to just throw more spindles and RAID10 everything.  Buy some time and
continue to optimize things at the software level... :)

Otherwise you could probably figure out if you're hitting an IOP limit
on your drive setup, saturating the IO path or dealing with CPU
limitations stemming from parity calculations... sometimes those
hardware RAID controllers are actually slower than software RAID due to
their piddly processors (I don't know what controller you have).

Is write caching enabled (and I hope you have battery backup on the
controller if it is)?  This helps a LOT w/ RAID4/5/6 as you're probably
aware... :)

> 
> > Is the disk array you bought an N-series? (N3300, N3600)?  If so, those
> > are NetApps and should be quite fast thanks to heavy write caching.
> > Even then, you'll be limited by spindles it sounds like...
> 
> No, the array is IBM brand.   EXP3000 (I've since looked it up)
> 

Ah, should be pretty decent.  The EXP we have has write caching as
well -- however ours is all FC drives.  The larger SATA drives might
work out fine if you have a lot of spindles and are doing RAID10... I
don't like doing RAID5 much with SATA though as the rebuild times are
huge and drives tend to fail more than the SAS resulting in disastrous
times for all :P

Interested to know how things turn out for you.

Ray