[CentOS-virt] io writes very slow when using vmware server

Mon Oct 6 08:39:12 UTC 2008
Pasi Kärkkäinen <pasik at iki.fi>

On Sun, Oct 05, 2008 at 11:28:45AM +0200, Paul Bijnens wrote:
> We are struggling with a strange problem.
> When we have some VMWare clients running (mostly MS windows clients),
> than the IO-write performance on the host becomes very bad.
> The guest os's do not do anything, just having them started,
> sitting at the login prompt, is enough to trigger the problem.
> The host has plenty of 4G of RAM, and all clients fit easily into
> the space.
> The disksystem is a RAID5 (ok, not the fastest choice).
> Both hosts are running the latest CentOS 5.2 (but the problem was
> already present in 5.0 as well), and running the latest production
> version of VMWare server 1.0.7 (but when running 1.0.5 the problem
> was there too).


I assume you know VMware server v1.x is not supported on RHEL5/CentOS5.. 

Have you tried VMware server v2.0?

> One host as an adaptec 2820 with 4 SATA drives, and uses the aacraid
> driver. Another host is a proliant with a smart array E200, battery
> backed cache of 128 MB using the cciss driver.
> Filesystem is ext3 on both hosts.
> When measuring the write performance, I try to avoid measuring the
> use the buffer cache like this, writing a 4.2gb file:
>   $ sync; time sh -c 'dd if=/dev/zero of=large bs=8k count=500k; sync'
> (Ignore the times that dd reports because those are including the buffer
> cache; divide 4.2gb by the number of seconds reported by "time".)
> To verify, I also open a another terminal and run there:
>   $ iostat -kx /dev/sda 5 1000
> On my own workstation with a single sata drive, I get about 54 MB/s.
> On the hosts having the RAID5 disks, and without any vmware client
> running, I get about 20MB/s. As said, not really fast, but it should be
> good enough for our purpose.

20 MB/sec sounds really bad. There has to be something wrong with your
raid/disk setup.

Any errors/warnings in dmesg? or syslog/messages? 

> However when I start one vmclient and repeat the test, the write speed
> drops to half to about 10MB/s. With 2 clients the write speed drops even
> more to about 5 MB/s. With 3 clients only 2 MB/s, and with 4 MB/s
> I get only 0.5-1 MB/s write speed.

Even without any disk activity on VMware guests? 

> The iostat shows the disk is 100% utilized, and the processor is 
> spending most of it's time (99-100%!) in iowait.

This is the problem. For some reason the raid is performing really poorly
and causing a lot of iowait. 

> Here is some output of iostat on an actually "idle" machine,
> running 3 MS Windows clients, all guest OS's are just sitting doing 
> almost nothing, not even a antivirus program installed:
> Has anyone ever seen this before?
> Any tips to find the cause?

I haven't really seen this myself.. I'd start with fixing the host OS disk
performance first before installing VMware server at all.

-- Pasi