On Sun, Oct 05, 2008 at 11:28:45AM +0200, Paul Bijnens wrote: > > We are struggling with a strange problem. > When we have some VMWare clients running (mostly MS windows clients), > than the IO-write performance on the host becomes very bad. > The guest os's do not do anything, just having them started, > sitting at the login prompt, is enough to trigger the problem. > > The host has plenty of 4G of RAM, and all clients fit easily into > the space. > The disksystem is a RAID5 (ok, not the fastest choice). > > Both hosts are running the latest CentOS 5.2 (but the problem was > already present in 5.0 as well), and running the latest production > version of VMWare server 1.0.7 (but when running 1.0.5 the problem > was there too). > Hello. I assume you know VMware server v1.x is not supported on RHEL5/CentOS5.. Have you tried VMware server v2.0? > One host as an adaptec 2820 with 4 SATA drives, and uses the aacraid > driver. Another host is a proliant with a smart array E200, battery > backed cache of 128 MB using the cciss driver. > Filesystem is ext3 on both hosts. > > > When measuring the write performance, I try to avoid measuring the > use the buffer cache like this, writing a 4.2gb file: > > $ sync; time sh -c 'dd if=/dev/zero of=large bs=8k count=500k; sync' > > (Ignore the times that dd reports because those are including the buffer > cache; divide 4.2gb by the number of seconds reported by "time".) > > To verify, I also open a another terminal and run there: > > $ iostat -kx /dev/sda 5 1000 > > On my own workstation with a single sata drive, I get about 54 MB/s. > > On the hosts having the RAID5 disks, and without any vmware client > running, I get about 20MB/s. As said, not really fast, but it should be > good enough for our purpose. > 20 MB/sec sounds really bad. There has to be something wrong with your raid/disk setup. Any errors/warnings in dmesg? or syslog/messages? > However when I start one vmclient and repeat the test, the write speed > drops to half to about 10MB/s. With 2 clients the write speed drops even > more to about 5 MB/s. With 3 clients only 2 MB/s, and with 4 MB/s > I get only 0.5-1 MB/s write speed. > Even without any disk activity on VMware guests? > The iostat shows the disk is 100% utilized, and the processor is > spending most of it's time (99-100%!) in iowait. > This is the problem. For some reason the raid is performing really poorly and causing a lot of iowait. > Here is some output of iostat on an actually "idle" machine, > running 3 MS Windows clients, all guest OS's are just sitting doing > almost nothing, not even a antivirus program installed: > > > Has anyone ever seen this before? > Any tips to find the cause? > I haven't really seen this myself.. I'd start with fixing the host OS disk performance first before installing VMware server at all. -- Pasi