[CentOS-virt] CentOS 6 kvm disk write performance

I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, disk write performance in kvm guest VMs is much worse.

There are many, many posts about optimising kvm, many mentioning disk performance in CentOS 5 vs 6. I've tried various changes to speed up write performance, but northing's made a significant difference so far:

- Install virtio disk drivers in guest - update the host software - Update RAID firmware to latest version - Switch the host disk scheduler to deadline - Increase host RAM from 8GB to 24GB - Increase guest RAM from 2GB to 4GB - Try different kvm cache options - Switch host from ext4 back to ext3 - Set noatime on the virtual disk image file Note: There is no encryption or on-access virus scanner on any host or guest.

Below are some the block write figures in MB/s from bonnie++ with various configurations:

First, figures for the hosts show that the CentOS 6 server is faster:

54 CentOS 5 Host 50 CentOS 5 Host 69 CentOS 6 host 70 CentOS 6 host

Figures for a CentOS 6 guest running on the CentOS 5 host show that the performance hit is less than 50%:

30 CentOS 6 guest on CentOS 5 host with no optimisations 27 CentOS 6 guest on CentOS 5 host with no optimisations 32 CentOS 6 guest on CentOS 5 host with no optimisations

Here are the figures a CentOS 6 guest running on the CentOS 6 host with various optimisations. Even with these optimisations, performance doesn't come close to the un-optimised guest running on the CentoOS 5 host:

5 No optimisations (i.e. same configuration as on CentOS 5) 4 deadline scheduler 5 deadline scheduler 15 noatime,nodiratime 14 noatime,nodiratime 15 noatime 15 noatime + deadline scheduler 13 virtio 13 virtio 10 virtio + noatime 9 virtio + noatime

The CentOS 6 server has a better RAID card, different disks and more RAM, which might account for the better CentOS 6 host performance. But why might the guest write performance be so much worse?

Is this a known problem? If so, what's the cause? If not, is there a way to locate the problem rather than using trial and error?

Thanks, Julian