Really bad KVM disk performance - Discuss

20 Feb 2012


      Hi Gang,
I recently rented a server at a datacenter with Centos 5.7 X64, Q9550
Processor, 8GB Ram, and dual 250GB SATA HDs (with 16mb cache).  They had
loaded it with KVM, and installed a 30-day trial of Virtualizor as the
front-end for KVM.
I was so impressed with how fasts the guests ran that I want to build a few of
these machines for myself.  I just installed one: same Q9550 processor, 4GB
ram, and dual 250GB SATA HDs (with 32mb cache).  I installed Centos 6.2 X64,
and installed Webmin's Cloudmin as the front-end.
Immediately when I was installing stuff, I could tell this new system I just
built was not nearly as fast as the first one.  I ran some CPU and disk
benchmarking programs, and saw that while the CPU stuff tested similarly, the
disk thruput was much different... Down-right poor in one of the guests!
On both systems, /dev/md2 is a LVM reserved exclusively for KVM guests.  So
each guest is running in its own logical volume, in software raid.
Thinking there may be something wrong with the HDs, I ran Bonnie (
http://www.coker.com.au/bonnie++/ ) and compared both host machines.  They
tested fairly similar (within 10%).  Yet comparing their guests is like night
and day. Example:
On good machine's Centos 5.7 x32 guest install:
# hdparm -tT /dev/hda
/dev/hda:
 Timing cached reads:   26760 MB in  1.99 seconds = 13417.10 MB/sec
 Timing buffered disk reads:  388 MB in  3.01 seconds = 128.86 MB/sec
On my machine's Centos 5.7 x32 guest install:
# hdparm -tT /dev/hda
/dev/hda:
 Timing cached reads:   1864 MB in  2.16 seconds = 863.87 MB/sec
 Timing buffered disk reads:  358 MB in  3.08 seconds = 116.17 MB/sec
On one of my machine's Mandrake 8.2 x32 guest install:
# hdparm -tT /dev/hda
/dev/hda:
 Timing buffer-cache reads:   27000 MB in  2.00 seconds = 13500.00 MB/sec
 Timing buffered disk reads:   12 MB in  3.66 seconds =   3.28 MB/sec
On that system, the hdparm's -i output shows:
# hdparm -i /dev/hda
/dev/hda:
Model=QEMU HARDDISK, FwRev=0.12.1, SerialNo=QM00001
 Config={ Fixed }
 RawCHS=16383/16/63, TrkSize=32256, SectSize=512, ECCbytes=4
 BuffType=DualPortCache, BuffSize=256kB, MaxMultSect=16, MultSect=16
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=73400320
 IORDY=yes, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2
 DMA modes:  sdma0 sdma1 sdma2 mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
 AdvancedPM=no
 Drive conforms to: ATA/ATAPI-5 published, ANSI NCITS 340-2000:
* signifies the current active mode
The bonnie numbers show for sequential output:
Good Machine Host: 76,857K/Sec
My Machine Host: 72,561K/Sec
Good Machine Centos 5.7 Guest: 66,266K/sec
My Machine Centos 5.7 Guest: 20,623K/sec
My machine Mandrake Guest: 1,365K/sec
Where should I look?  I realize I do have two different front-ends to KVM, and
perhaps they are passing different parameters to it.  I am also running the
KVM from Centos 6.2 on my machine, vs the other server is running on 5.7, but
I would have thought that "newer is better".  Also note that my hard drives
have a larger cache.
On a side note, I'm not thrilled with the Virtualizor's tech support, but the
product seems easy to use, once it actually works.  Cloudmin seems to be
buggy, and not let you do things like change cd images on the fly, access the
console before the machine fully boots (!)... Any suggestions on other,
preferably open-source options?  I'm a definite newbie to this virtualization
stuff.
Bob