<div dir="ltr">On Wed, Mar 10, 2010 at 5:28 PM, Steven Ellis <span dir="ltr"><<a href="mailto:steven.ellis@bulletin.net">steven.ellis@bulletin.net</a>></span> wrote:<br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div bgcolor="#ffffff" text="#000000">
<br>
Is anyone else having major I/O peaks due to logrotate or other jobs
running simultaneously across multiple guests. I have one KVM server
running Centos 5.4 with local disk that is seriously suffering as most
of the guests rotate their syslog at the same time.<br>
<br>
Looking at the KVM server I'm seeing<br>
<br>
11:00:01 PM CPU %user %nice %system %iowait
%steal %idle<br>
03:40:01 AM all 0.07 0.00 2.74 0.93
0.00 96.26<br>
03:50:01 AM all 0.07 0.00 1.17 1.18
0.00 97.58<br>
04:00:01 AM all 0.08 0.00 1.51 0.82
0.00 97.59<br>
04:10:02 AM all 0.53 0.03 15.31 51.61
0.00 32.53<br>
04:20:01 AM all 0.28 0.12 4.12 22.21
0.00 73.27<br>
04:30:01 AM all 0.07 0.00 0.80 1.21
0.00 97.92<br>
04:40:01 AM all 0.07 0.00 2.60 1.81
0.00 95.52<br>
04:50:01 AM all 0.08 0.00 0.79 1.44
0.00 97.69<br>
<br>
On one of the guests running Centos 4.6 the impact is so bad I get DMA
timeout errors in the syslog, and occasional kernel panics.<br>
<br>
Mar 11 04:05:04 localhost kernel: hda: dma_timer_expiry: dma status ==
0x21<br>
Mar 11 04:05:14 localhost kernel: hda: DMA timeout error<br>
Mar 11 04:05:14 localhost kernel: hda: dma timeout error: status=0x50 {
DriveReady SeekComplete }<br>
Mar 11 04:05:14 localhost kernel: <br>
Mar 11 04:05:14 localhost kernel: ide: failed opcode was: unknown<br>
Mar 11 04:05:59 localhost kernel: hda: dma_timer_expiry: dma status ==
0x21<br>
Mar 11 04:06:14 localhost kernel: hda: DMA timeout error<br>
Mar 11 04:06:14 localhost kernel: hda: dma timeout error: status=0x50 {
DriveReady SeekComplete }<br>
<br>
<br>
One reference I've found is at <br>
* <a href="http://lonesysadmin.net/linux-virtual-machine-tuning-guide/" target="_blank">http://lonesysadmin.net/linux-virtual-machine-tuning-guide/</a><br>
<br>
This suggests avoiding running scheduled jobs simultaneously across
guests, and suggests using a random sleep.<br>
<br>
Does anyone else have suggestions on reducing the impact of
cron/logrotate.<br><br></div></blockquote><div><br>I ran into this issue as well on a box running Xen with local
storage. <br>
<br>
My solution was to modify /etc/crontab to run /etc/cron.weekly at
different times for each guest and for the dom0. I modified the entry
on each VM to be 10 minutes after the previous one and have not seen any
load spikes since then. <br>
<br>
Matt<br clear="all"><br>--<br>Mathew S. McCarrell<br>Clarkson University
'10<br><br><a href="mailto:mccarrms@gmail.com">mccarrms@gmail.com</a><br><a href="mailto:mccarrms@clarkson.edu">mccarrms@clarkson.edu</a><br>1-518-314-9214 <br></div></div><br></div>