[CentOS] disk I/O problems and Solutions

Fri Oct 9 21:41:22 UTC 2009
Ross Walker <rswwalker at gmail.com>

On Fri, Oct 9, 2009 at 1:34 PM, Alan McKay <alan.mckay at gmail.com> wrote:
> On Fri, Oct 9, 2009 at 1:28 PM, Ray Van Dolson <rayvd at bludgeon.org> wrote:
>> Really hard to say what's going on.  Does your DB need optimization?
>> Do the applications hitting it?  Maybe some indexing?  Maybe some more
>> RAM on the machine would help?  What exactly is the workload like --
>> especially during the time when you're peaked out?
>
> Yes, these are all things we are just starting to look at - so no
> clear answers yet
>
>> Is the system swapping?  If so, you either need more memory or need to
>> track down a memory leak.... 'free' and 'sar' can both help you see
>> what swap usage is like.
>
> That's one thing it does not seem to be doing, fortunatley.
>
>> Writes are always slower on any parity based RAID setup, so I imagine
>> you'd get superior performance on RAID10, especially if you're write
>> heavy.
>
> We are generally read heavy but at certain times of day including 6 to
> 9am we have huge batches of writes.
>

I would relocate the log files to a LV on the two disk mirror for now
so the writes can batch to log then to database lazily without hitting
the database with the RAID5 write penalty.

Since your writes only come in during a 3 hour window in the morning
you don't necessarily need RAID10 for the whole thing, but the log
files need to be off the RAID5, either on RAID10, RAID1 or SSD.

Oh and make sure your VGs/LVs/partitions are set on the HW RAID
alignment boundary and if Postgres and the file system you use has
support for chunk size/stripe width you use it.

-Ross