We are struggling with a strange problem.
When we have some VMWare clients running (mostly MS windows clients),
than the IO-write performance on the host becomes very bad.
The guest os's do not do anything, just having them started,
sitting at the login prompt, is enough to trigger the problem.
The host has plenty of 4G of RAM, and all clients fit easily into
the space.
The disksystem is a RAID5 (ok, not the fastest choice).
Both hosts are running the latest CentOS 5.2 (but the problem was
already present in 5.0 as well), and running the latest production
version of VMWare server 1.0.7 (but when running 1.0.5 the problem
was there too).
One host as an adaptec 2820 with 4 SATA drives, and uses the aacraid
driver. Another host is a proliant with a smart array E200, battery
backed cache of 128 MB using the cciss driver.
Filesystem is ext3 on both hosts.
When measuring the write performance, I try to avoid measuring the
use the buffer cache like this, writing a 4.2gb file:
$ sync; time sh -c 'dd if=/dev/zero of=large bs=8k count=500k; sync'
(Ignore the times that dd reports because those are including the buffer
cache; divide 4.2gb by the number of seconds reported by "time".)
To verify, I also open a another terminal and run there:
$ iostat -kx /dev/sda 5 1000
On my own workstation with a single sata drive, I get about 54 MB/s.
On the hosts having the RAID5 disks, and without any vmware client
running, I get about 20MB/s. As said, not really fast, but it should be
good enough for our purpose.
However when I start one vmclient and repeat the test, the write speed
drops to half to about 10MB/s. With 2 clients the write speed drops even
more to about 5 MB/s. With 3 clients only 2 MB/s, and with 4 MB/s
I get only 0.5-1 MB/s write speed.
The iostat shows the disk is 100% utilized, and the processor is
spending most of it's time (99-100%!) in iowait.
Here is some output of iostat on an actually "idle" machine,
running 3 MS Windows clients, all guest OS's are just sitting doing
almost nothing, not even a antivirus program installed:
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 33.80 0.60 76.80 2.40 443.20 11.51
110.46 1269.85 12.93 100.04
avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 1.50 75.98 0.00 22.42
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 75.00 0.20 86.40 0.80 630.40 14.58
105.74 1466.97 11.55 100.06
avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 8.09 14.89 0.00 76.92
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 27.00 0.60 32.60 2.40 346.40 21.01
19.10 1079.57 10.57 35.10
avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 2.50 16.12 0.00 81.28
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 82.40 0.40 24.80 1.60 285.60 22.79
26.31 430.37 12.52 31.54
avg-cpu: %user %nice %system %iowait %steal %idle
0.10 0.00 0.70 6.71 0.00 92.49
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz
avgqu-sz await svctm %util
sda 0.00 7.40 0.40 24.20 1.60 271.20 22.18
7.12 918.03 8.59 21.14
Has anyone ever seen this before?
Any tips to find the cause?
--
Paul
