[CentOS-virt] Logrotate/cron and major I/O contention with KVM.

[Mathew S. McCarrell]

On Wed, Mar 10, 2010 at 5:28 PM, Steven Ellis <steven.ellis at bulletin.net>wrote:

>
> Is anyone else having major I/O peaks due to logrotate or other jobs
> running simultaneously across multiple guests. I have one KVM server running
> Centos 5.4 with local disk that is seriously suffering as most of the guests
> rotate their syslog at the same time.
>
> Looking at the KVM server I'm seeing
>
> 11:00:01 PM       CPU     %user     %nice   %system   %iowait    %steal
> %idle
> 03:40:01 AM       all      0.07      0.00      2.74      0.93      0.00
> 96.26
> 03:50:01 AM       all      0.07      0.00      1.17      1.18      0.00
> 97.58
> 04:00:01 AM       all      0.08      0.00      1.51      0.82      0.00
> 97.59
> 04:10:02 AM       all      0.53      0.03     15.31     51.61      0.00
> 32.53
> 04:20:01 AM       all      0.28      0.12      4.12     22.21      0.00
> 73.27
> 04:30:01 AM       all      0.07      0.00      0.80      1.21      0.00
> 97.92
> 04:40:01 AM       all      0.07      0.00      2.60      1.81      0.00
> 95.52
> 04:50:01 AM       all      0.08      0.00      0.79      1.44      0.00
> 97.69
>
> On one of the guests running Centos 4.6 the impact is so bad I get DMA
> timeout errors in the syslog, and occasional kernel panics.
>
> Mar 11 04:05:04 localhost kernel: hda: dma_timer_expiry: dma status == 0x21
> Mar 11 04:05:14 localhost kernel: hda: DMA timeout error
> Mar 11 04:05:14 localhost kernel: hda: dma timeout error: status=0x50 {
> DriveReady SeekComplete }
> Mar 11 04:05:14 localhost kernel:
> Mar 11 04:05:14 localhost kernel: ide: failed opcode was: unknown
> Mar 11 04:05:59 localhost kernel: hda: dma_timer_expiry: dma status == 0x21
> Mar 11 04:06:14 localhost kernel: hda: DMA timeout error
> Mar 11 04:06:14 localhost kernel: hda: dma timeout error: status=0x50 {
> DriveReady SeekComplete }
>
>
> One reference I've found is at
>  * http://lonesysadmin.net/linux-virtual-machine-tuning-guide/
>
> This suggests avoiding running scheduled jobs simultaneously across guests,
> and suggests using a random sleep.
>
> Does anyone else have suggestions on reducing the impact of cron/logrotate.
>
>
I ran into this issue as well on a box running Xen with local storage.

My solution was to modify /etc/crontab to run /etc/cron.weekly at different
times for each guest and for the dom0.  I modified the entry on each VM to
be 10 minutes after the previous one and have not seen any load spikes since
then.

Matt

--
Mathew S. McCarrell
Clarkson University '10

mccarrms at gmail.com
mccarrms at clarkson.edu
1-518-314-9214
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.centos.org/pipermail/centos-virt/attachments/20100311/b5ebd72d/attachment.html