[CentOS] IO causing major performance issues

Thu Nov 15 23:06:52 UTC 2007
Antonio Varni <avarni at estalea.com>

Hello everyone.

I'm wondering what other people's experiences are WRT systems becoming
unresponsive (unable to ssh in, etc) for brief periods of time when
a large amount of IO is being performed.  It's really starting to
cause a problem for us.  We're on Dell PowerEdge 1955 blades - but this same
issue has caused us problems on PE1950, PE1850, PE1750 servers.

We're running Centos 4.5 right now. I know Centos 5 includes ionice, more
io scheduler/elevator selections like deadlock/etc. Perhaps that would
fix this issue.  We're running the latest PERC firmware.

The specific issue I'm referring to at this point is on a system running
mysql. All mysql data files are on a netapp filer but mysql's tmp directory
is on local disk.  Whenever a lot of temp tables are created (and thus
written and deleted from local disk quickly) we can't even log in to the
machine - and our monitoring system gets all freaked out and we get
lots of pages, etc... FYI this is two disks with hardware raid 1.

Is it just me? Or is this specific to Dell systems, or is this just
the state of the Linux kernel these days? Is there some magical patch
I can apply to make this issue go away :)


Thanks in advance for any insight into this issue.

Antonio



-- 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
antonio varni
[ technology ]

ESTALEA, L.P.
629 State Street #222
Santa Barbara, CA 93101
v 805.252.0115
f 805.899.2697
e avarni at estalea.com
w www.estalea.com