[CentOS] disk I/O problems and Solutions

Fri Oct 9 17:34:17 UTC 2009
Alan McKay <alan.mckay at gmail.com>

On Fri, Oct 9, 2009 at 1:28 PM, Ray Van Dolson <rayvd at bludgeon.org> wrote:
> Really hard to say what's going on.  Does your DB need optimization?
> Do the applications hitting it?  Maybe some indexing?  Maybe some more
> RAM on the machine would help?  What exactly is the workload like --
> especially during the time when you're peaked out?

Yes, these are all things we are just starting to look at - so no
clear answers yet

> Is the system swapping?  If so, you either need more memory or need to
> track down a memory leak.... 'free' and 'sar' can both help you see
> what swap usage is like.

That's one thing it does not seem to be doing, fortunatley.

> Writes are always slower on any parity based RAID setup, so I imagine
> you'd get superior performance on RAID10, especially if you're write
> heavy.

We are generally read heavy but at certain times of day including 6 to
9am we have huge batches of writes.

> But to begin with, it'd be interesting to know exactly what this server
> is doing.  Does it makes sense that the disks are being brought to
> their knees with the given workload?

Oh yes!   We are a financial services provider and the Db contains all
of the worlds stock prices for about the last 10 years :-)  And every
day new batches of prices come in (at different times depending on
markets).   This system is just a PostgreSQL DB.   Nothing else.

> Is the disk array you bought an N-series? (N3300, N3600)?  If so, those
> are NetApps and should be quite fast thanks to heavy write caching.
> Even then, you'll be limited by spindles it sounds like...

No, the array is IBM brand.   EXP3000 (I've since looked it up)


-- 
“Don't eat anything you've ever seen advertised on TV”
         - Michael Pollan, author of "In Defense of Food"