Well the CFQ scheduler tries to do just that, but if the amount of io is overwhelming even it cannot compensate. Writes take longer then reads and for a mirror they have to happen on both drives before they can service another request. I suggest you put the temp database on a tempfs to avoid the problem. Set cfq scheduler on /dev/sda if it isn't already. Other then that you could try a different file system like xfs or jfs to see if that helps. -Ross -----Original Message----- From: centos-bounces at centos.org <centos-bounces at centos.org> To: CentOS mailing list <centos at centos.org> Sent: Thu Nov 15 19:29:20 2007 Subject: RE: [CentOS] IO causing major performance issues Of course IO can swamp the file system. My point is that the kernel should at least give enough time-slices to the other processes (like sshd) so we can still log in. It's not asking a lot from the kernel - to just log in via ssh really. -- _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ antonio varni [ technology ] ESTALEA, L.P. 629 State Street #222 Santa Barbara, CA 93101 v 805.252.0115 f 805.899.2697 e avarni at estalea.com w www.estalea.com On Thu, 15 Nov 2007, Ross S. W. Walker wrote: > redhat at mckerrs.net wrote: > > Antonio Varni wrote: > > > > > > Hello everyone. > > > > > > I'm wondering what other people's experiences are WRT systems becoming > > > unresponsive (unable to ssh in, etc) for brief periods of time when > > > a large amount of IO is being performed. It's really starting to > > > cause a problem for us. We're on Dell PowerEdge 1955 blades > > > - but this same > > > issue has caused us problems on PE1950, PE1850, PE1750 servers. > > > > > > We're running Centos 4.5 right now. I know Centos 5 includes > > > ionice, more > > > io scheduler/elevator selections like deadlock/etc. Perhaps that would > > > fix this issue. We're running the latest PERC firmware. > > > > > > The specific issue I'm referring to at this point is on a > > > system running > > > mysql. All mysql data files are on a netapp filer but mysql's > > > tmp directory > > > is on local disk. Whenever a lot of temp tables are created (and thus > > > written and deleted from local disk quickly) we can't even > > > log in to the > > > machine - and our monitoring system gets all freaked out and we get > > > lots of pages, etc... FYI this is two disks with hardware raid 1. > > > > > > Is it just me? Or is this specific to Dell systems, or is this just > > > the state of the Linux kernel these days? Is there some magical patch > > > I can apply to make this issue go away :) > > > > > > > > > Thanks in advance for any insight into this issue. > > > > > > Antonio > > > > I have noticed similar behaviour on all sort of linuxes (in > > particular, ssh into the box is really slow when it's doing > > IO) and wondered why, but never really thought about > > investigating any further. > > > > Unfortunately, I do a lot of work with solaris and the funny > > thing is that I have *never* seen a solaris kernel exhibit > > this sort of behaviour. Even if it is installed on normal > > IDE/SATA disks. And, in fact, even if installed on the exact > > same hardware. > > > > > > Now I'm curious.....especially given that I'm right in the > > middle of pushing to get rid of solaris in favour of RHEL. > > It really depends what the system is doing, what services you are > running and how you have it configured. > > You had Solaris installed, what services was it running? > > You had Linux installed, what services was it running? > > Database temp tables and logs can generate an enormous amount of > io which can swamp the file systems of any system. > > I have seen it on Windows and Linux, so I don't see why Solaris > would be any different. > > You could always try a different scheduler to see if that helps, > for instance if you are using 'cfq' try 'deadline'. > > -Ross > > ______________________________________________________________________ > This e-mail, and any attachments thereto, is intended only for use by > the addressee(s) named herein and may contain legally privileged > and/or confidential information. If you are not the intended recipient > of this e-mail, you are hereby notified that any dissemination, > distribution or copying of this e-mail, and any attachments thereto, > is strictly prohibited. If you have received this e-mail in error, > please immediately notify the sender and permanently delete the > original and any copy or printout thereof. > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > _______________________________________________ CentOS mailing list CentOS at centos.org http://lists.centos.org/mailman/listinfo/centos ______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20071115/9646bff7/attachment-0005.html>