On a 32bit Windows 2008 Server guest VM on a CentOS 5 host, iometer reported a disk write speed of 37MB/s
The same VM on a CentOS 6 host reported 0.3MB/s. i.e. The VM was unusable.
Write performance in a CentOS 6 VM was also much worse, but it was usable. (See http://lists.centos.org/pipermail/centos-virt/2012-August/002961.html)
With iometer still running in the guest, I installed tuned on the CentOS 6 host and enabled the virtual-host profile. iometer showed the average disk write speed increasing immediately. A fresh iometer test reported a disk write speed of 80MB/s.
I'm not sure if the tuned-adm virtual-host profile was available in 6.0 - it may be necessary to update to get it. http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/6/html/Power_Mana...
Installation is quick & easy: yum install tuned tuned-adm profile virtual-host
I'm surprised not to see any other posts on this subject. Maybe there's something peculiar about my setup that caused the dreadful write performance & tuned-adm happened to fix it. This problem has been a real headache and I'm extremely grateful to Philip Durbin and jimi_c for bringing the solution to my attention.
The host server was rebuilt from scratch after a RAID1 simultaneous double disk failure (happily my DR strategy worked. Have you tested yours recently?). It appears that CentOS 6 has some different defaults (more conservative power saving?). Maybe upgrades don't change the defaults, so this issue only affects new CentOS 6 installs?
The iometer settings were: 1000 sectors, 16 outstanding IOs, access specification: 16k, 0%read 0%random The figures I've reported above are the 'Total MBs per second' See http://www.iometer.org/doc/downloads.html
In the process of researching the problem, 2 other quick & easy optimisations came to light, each adding a few more MB/s: - Set noatime. See: http://www.activoinc.com/blog/2009/08/25/setting-noatime-and-nodiratime-for-...
- Disable Windows disk cacheing: See http://pve.proxmox.com/wiki/Windows_2003_guest_best_practices On Windows 2008: Computer Mgmt -> Disk Mgmt -> Select Drive (system) -> properties -> Policies tab -> uncheck 'Optimize for quick removal'
I'm surprised not to see any other posts on this subject. Maybe there's something peculiar about my setup that caused the dreadful write performance & tuned-adm happened to fix it. This problem has been a real headache and I'm extremely grateful to Philip Durbin and jimi_c for bringing the solution to my attention.
I don't know about 6.0 because I went from 5.something to 6.2, then 6.3 and tuned was installed by default but not with those profiles. With CentOS 6.2, I manually tuned things, played with read-ahead sizes, etc, until I got them optimized. Your milage may vary but with a machine with 32GB of RAM, I got the best performace with a read ahead of 16384 blocks. I set the I/O schedualar to deadline to limit latency. I played with various cache modes and for most of my applications write-back was okay (i.e., not storing critical financial data and crashes are few and far between, journalling file systems are used, so worst case someone might lose a couple minutes work).
But that said, one weirdness I have run into while playing with options to get latency to a minimum, I found that if I enabled kernel premption, the kernel would only use one core. Is there anyway to have a preemptive kernel on a multicore machine and still use all the cores?
-
Julian price -
Nanook