On 10.08.2012 13:46, Julian price wrote:

...

I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6, disk write performance in kvm guest VMs is much worse. ... Is this a known problem? If so, what's the cause? If not, is there a way to locate the problem rather than using trial and error?

Thanks, Julian

Nux! wrote:

How are you running KVM machines? Through libvirt?

Not through libvirt, but using the qemu-kvm command. This is the full command-line (without libvirt):

/usr/libexec/qemu-kvm -k en-gb -m 4096 -smp 2 -pidfile /media/archive/vm022/.vm022.pid -monitor unix:/media/archive/vm022/.vm022.monitor,server,nowait -usb -usbdevice tablet -net nic,macaddr=02:00:01:00:24:02 -net tap,script=/etc/incharge/qemu-ifup -vnc :2 -drive file=/media/vm022/hda.raw,index=0,media=disk -drive file=/media/archive/vm022/hdb.raw,index=1,media=disk -drive file=/media/archive/vm022/hdc.raw,index=2,media=disk

Have you made any changes to the cgroups?

No, I never heard of cgroups until today!

What are you using as disks, raw files, lvm

There are 2 physical disks in raid 1 configuration Apart from system and swap partitions, the majority of the disk is LVM Each VM has 3 virtual disk files. 1 system disk (hda), one temp disk (hdb) and one spare (hdc). All are raw format. The hda for each VM is stored in its own LVM volume, so a snapshot can be made and mirrored with minimal overhead. So the volume /media/vm022 contains one file (hda.raw) and one folder which is used by a monitoring task to check the volume hasn't gone read-only. hdb, hdc and the monitor & pid files are stored on a larger LVM volume /media/archive There was no LVM snapshot while any of the test timings were made.

As there are so few files on /media/vm022, the reserved space is reduced using tune2fs -r 73134 /dev/mapper/vm-ic022 ...so the 32GB file fits into a 33GB LVM volume. (See https://www.centos.org/modules/newbb/viewtopic.php?topic_id=30854)

All the above is identical on both the CentOS 5 and CentOS 6 servers, except that ext3 is used on the CentOS 5 host and ext4 on the CentOS 6 host, and the disk & raid hardware is a different brand. In all cases, the guest VM is CentOS 6.2 and uses ext4.

...

I have 2 similar servers. Since upgrading one from CentOS 5.5 to 6,

disk write performance in kvm guest VMs is much worse.

Philip Durbin wrote:

Nice post, Julian. It generated some feedback at

http://irclog.perlgeek.de/crimsonfu/2012-08-10 and a link to http://rhsummit.files.wordpress.com/2012/03/wagner_network_perf.pdf

Phil

Thanks Phil for linking to my post on #crimsonfu, and reporting the result back here.

In response to the points jimi_c raised there:

I don't see his test numbers for different caching options

All the test figures I gave are using cache=writethrough. cache=writeback produced much better figures than even the host (about 180MB/s), because it is really writing to memory, but I don't think it's a safe option to use. cache=none produced worse figures. I didn't include the figures because I did those tests before I started using bonnie++ (I was just timing copying files before that) and I'd already ruled cacheing out as a solution.

also, is he doing deadline on the guest, host, or both?

deadline on the host - didn't try it on the guest.

not sure if they implemented it yet, but they were talking about a vm

host setting for tuned

and one for guests yeah, Wagner mentioned it in his summit presentation:

http://rhsummit.files.wordpres%5B%E2%80%A6%5D_network_perf.pdf

they should be available in rhel 6.3 according to his presentation

Well, tuned-adm is a gift for part-time sysadmins like myself.

Some of the guest disk write figures were close to the host's & better than CentOS 5 after doing...

yum install tuned tuned-adm profile virtual-host

..in the host and...

yum install tuned tuned-adm profile virtual-guest

...in the guest.

Here are the new bonnie++ guest block write figures in MB/s. all using tuned-host and virtio, with & without tuned-guest. Not sure why there's so much variation, but at least they're all much better.

45 tuned-host 73 tuned-host 50 tuned-host + tuned-guest 37 tuned-host + tuned-guest

rhel/centos 6 probably went with a more conservative tuning option

Certainly looks that way. It's be interesting to know what & why.

Before jimi_c provided the tuned-adm tip, I was hoping that running the VM off a block device might be the answer. i.e: qemu-img convert /media/vm027/hda.raw -O /dev/vm/vm031

...but they are worse than running off a raw virtual disk file.

16 tuned-host virtio 20 tuned-host virtio 27 tuned-host tuned-guest virtio 24 tuned-host tuned-guest virtio

I'm not convinced, maybe there are other factors at work. I'd investigate further if my plan A wasn't back on track.

The bonnie++ figures I gave before measuring host disk write performance were for the host *root* partition, not the LVM volumes that the guest VMs use. What if the problem is LVM, not KVM? So I did some timings comparing the root drive with LVM volumes, some with & some without tuned-host. 'archive' and 'vmxxx' are the lvm volume names: (Note: these timings are done in the host, not in a guest)

69 / 69 / 70 / 65 / + tuned-host 64 / + tuned-host

55 archive 56 archive 53 vm027 48 vm027 33 vm027 + tuned-host 38 vm027 + tuned-host 53 vm022 + tuned-host 85 vm022 + tuned-host

This indicates that there is a wide variation in performance between different LVM volumes, and all the LVM volumes are performing worse than the root. (It's interesting that with tuned-host, the times seem to be mostly worse, but with greater deviation.) I repeated the above test on the CentOS 5 server (without tuned-host of course) and found the same thing - LVMs perform worse than root and vary widely:

54 / 50 / 39 archive 45 archive 39 archive 49 vm022 34 vm027 33 vm027

A slight performance hit might be expected for LVM, but I though it was meant to be negligible? If the figures fell into 2 bands - good and bad - then I'd be looking for a specific problem like a sector alignment, but they don't, and isn't sector alignment meant to be fixed on CentOS 6? The variation in performance indicates a problem of variable severity like fragmentation or the position on the physical disk - but I don't think either of those are likely causes, because there's only one file in each volume, and physical disk position shouldn't have such a marked effect should it? Any other suggestions?

Thanks, Julian

...

physical disk position shouldn't have such a marked effect should it?

Nanook wrote:

Actually the physical disk position can make a HUGE difference.

Thank you Nanook for your explanation. I think you're right. It looks like the variation in performance is not due to LVM, but the position on the disk. If the volumes' write performance figures are rearranged in the order of their position on the disk, they are roughly in line.

The root disk is first, so it is best. Then there are no figures for the swap file. The archive volume is roughly half the remaining space, so I'd expect it's performance to be closer to root. But archive is quite full, so maybe the bonnie++ test files went towards the end, near the other VM partitions. The last volume to be allocated has the worst performance. It all makes sense!

So, when calculating the disk performance hit from using a VM, or the effectiveness of VM disk optimisations, the benchmark is the performance of the volume that hosts the corresponding virtual disk, not the root disk.

For those reasons I always try to put swap and I/O critical stuff, like swap, at the beginning of the drive and loath partitioning software that thinks it's smarter and puts things where it wants.

And when there are many VMs, each with its own root disk, swap file, temp disk, there's a challenge! Well, at least I can try to ensure that the busiest VM gets the first logical volume.

All this shows the importance of basing decisions on evidence from performance tests on your own servers, not assumptions & other peoples' findings.

Thanks, Julian