[CentOS] IO causing major performance issues

Antonio Varni wrote:
> 
> Hello everyone.
> 
> I'm wondering what other people's experiences are WRT systems becoming
> unresponsive (unable to ssh in, etc) for brief periods of time when
> a large amount of IO is being performed.  It's really starting to
> cause a problem for us.  We're on Dell PowerEdge 1955 blades 
> - but this same
> issue has caused us problems on PE1950, PE1850, PE1750 servers.
> 
> We're running Centos 4.5 right now. I know Centos 5 includes 
> ionice, more
> io scheduler/elevator selections like deadlock/etc. Perhaps that would
> fix this issue.  We're running the latest PERC firmware.
> 
> The specific issue I'm referring to at this point is on a 
> system running
> mysql. All mysql data files are on a netapp filer but mysql's 
> tmp directory
> is on local disk.  Whenever a lot of temp tables are created (and thus
> written and deleted from local disk quickly) we can't even 
> log in to the
> machine - and our monitoring system gets all freaked out and we get
> lots of pages, etc... FYI this is two disks with hardware raid 1.
> 
> Is it just me? Or is this specific to Dell systems, or is this just
> the state of the Linux kernel these days? Is there some magical patch
> I can apply to make this issue go away :)
> 
> 
> Thanks in advance for any insight into this issue.

Yes, IO starvation can occur under heavy load.

Don't put database temp tables on system disks (or data tables for that matter).

How much memory do you have in this box and how big does the temp directory
usage get?

Why I ask is you could create a tempfs and have mysql use that, just make sure
you have enough memory that you can spare X (whatever your temp table usage is)
for a cache filesystem.

You would also notice a dramatic speed increase in MySQL.

-Ross

______________________________________________________________________
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.