<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7652.24">
<TITLE>Re: [CentOS] IO causing major performance issues</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<BR>
<P><FONT SIZE=2>Well the CFQ scheduler tries to do just that, but if the amount of io is overwhelming even it cannot compensate.<BR>
<BR>
Writes take longer then reads and for a mirror they have to happen on both drives before they can service another request.<BR>
<BR>
I suggest you put the temp database on a tempfs to avoid the problem.<BR>
<BR>
Set cfq scheduler on /dev/sda if it isn't already.<BR>
<BR>
Other then that you could try a different file system like xfs or jfs to see if that helps.<BR>
<BR>
-Ross<BR>
<BR>
<BR>
<BR>
-----Original Message-----<BR>
From: centos-bounces@centos.org <centos-bounces@centos.org><BR>
To: CentOS mailing list <centos@centos.org><BR>
Sent: Thu Nov 15 19:29:20 2007<BR>
Subject: RE: [CentOS] IO causing major performance issues<BR>
<BR>
Of course IO can swamp the file system. My point is that the kernel should<BR>
at least give enough time-slices to the other processes (like sshd) so<BR>
we can still log in. It's not asking a lot from the kernel - to just log in via ssh really.<BR>
<BR>
--<BR>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _<BR>
antonio varni<BR>
[ technology ]<BR>
<BR>
ESTALEA, L.P.<BR>
629 State Street #222<BR>
Santa Barbara, CA 93101<BR>
v 805.252.0115<BR>
f 805.899.2697<BR>
e avarni@estalea.com<BR>
w www.estalea.com<BR>
<BR>
On Thu, 15 Nov 2007, Ross S. W. Walker wrote:<BR>
<BR>
> redhat@mckerrs.net wrote:<BR>
> > Antonio Varni wrote:<BR>
> > ><BR>
> > > Hello everyone.<BR>
> > ><BR>
> > > I'm wondering what other people's experiences are WRT systems becoming<BR>
> > > unresponsive (unable to ssh in, etc) for brief periods of time when<BR>
> > > a large amount of IO is being performed. It's really starting to<BR>
> > > cause a problem for us. We're on Dell PowerEdge 1955 blades<BR>
> > > - but this same<BR>
> > > issue has caused us problems on PE1950, PE1850, PE1750 servers.<BR>
> > ><BR>
> > > We're running Centos 4.5 right now. I know Centos 5 includes<BR>
> > > ionice, more<BR>
> > > io scheduler/elevator selections like deadlock/etc. Perhaps that would<BR>
> > > fix this issue. We're running the latest PERC firmware.<BR>
> > ><BR>
> > > The specific issue I'm referring to at this point is on a<BR>
> > > system running<BR>
> > > mysql. All mysql data files are on a netapp filer but mysql's<BR>
> > > tmp directory<BR>
> > > is on local disk. Whenever a lot of temp tables are created (and thus<BR>
> > > written and deleted from local disk quickly) we can't even<BR>
> > > log in to the<BR>
> > > machine - and our monitoring system gets all freaked out and we get<BR>
> > > lots of pages, etc... FYI this is two disks with hardware raid 1.<BR>
> > ><BR>
> > > Is it just me? Or is this specific to Dell systems, or is this just<BR>
> > > the state of the Linux kernel these days? Is there some magical patch<BR>
> > > I can apply to make this issue go away :)<BR>
> > ><BR>
> > ><BR>
> > > Thanks in advance for any insight into this issue.<BR>
> > ><BR>
> > > Antonio<BR>
> ><BR>
> > I have noticed similar behaviour on all sort of linuxes (in<BR>
> > particular, ssh into the box is really slow when it's doing<BR>
> > IO) and wondered why, but never really thought about<BR>
> > investigating any further.<BR>
> ><BR>
> > Unfortunately, I do a lot of work with solaris and the funny<BR>
> > thing is that I have *never* seen a solaris kernel exhibit<BR>
> > this sort of behaviour. Even if it is installed on normal<BR>
> > IDE/SATA disks. And, in fact, even if installed on the exact<BR>
> > same hardware.<BR>
> ><BR>
> ><BR>
> > Now I'm curious.....especially given that I'm right in the<BR>
> > middle of pushing to get rid of solaris in favour of RHEL.<BR>
><BR>
> It really depends what the system is doing, what services you are<BR>
> running and how you have it configured.<BR>
><BR>
> You had Solaris installed, what services was it running?<BR>
><BR>
> You had Linux installed, what services was it running?<BR>
><BR>
> Database temp tables and logs can generate an enormous amount of<BR>
> io which can swamp the file systems of any system.<BR>
><BR>
> I have seen it on Windows and Linux, so I don't see why Solaris<BR>
> would be any different.<BR>
><BR>
> You could always try a different scheduler to see if that helps,<BR>
> for instance if you are using 'cfq' try 'deadline'.<BR>
><BR>
> -Ross<BR>
><BR>
> ______________________________________________________________________<BR>
> This e-mail, and any attachments thereto, is intended only for use by<BR>
> the addressee(s) named herein and may contain legally privileged<BR>
> and/or confidential information. If you are not the intended recipient<BR>
> of this e-mail, you are hereby notified that any dissemination,<BR>
> distribution or copying of this e-mail, and any attachments thereto,<BR>
> is strictly prohibited. If you have received this e-mail in error,<BR>
> please immediately notify the sender and permanently delete the<BR>
> original and any copy or printout thereof.<BR>
><BR>
> _______________________________________________<BR>
> CentOS mailing list<BR>
> CentOS@centos.org<BR>
> <A HREF="http://lists.centos.org/mailman/listinfo/centos">http://lists.centos.org/mailman/listinfo/centos</A><BR>
><BR>
_______________________________________________<BR>
CentOS mailing list<BR>
CentOS@centos.org<BR>
<A HREF="http://lists.centos.org/mailman/listinfo/centos">http://lists.centos.org/mailman/listinfo/centos</A><BR>
</FONT>
</P>
<P></P>
<HR WIDTH="100%">
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.
</BODY>
</HTML>