Hello all,
I have a PowerEdge 2950 that I am using to host Windows XP VM's under CentOS. I am experiencing poor performance in the Xen VM's. Any time I try to do something, the CPU usage (as reported by the Windows task manager) hits 100% for a few and then drops back down. This is causing the VM's to run sluggish and be quite annoying to use. Example.. I click IE.. cpu hits 100% and IE takes a bit to open. I go to a website.. 100%.. I click a link.. 100%. This is the case even if I only have one VM running.
I now have 32 VM's running on the machine and dom0's cpu usage as reported by xm top is 622%. It seems like Xen is just performing poorly while handling these Windows VMs. Any suggestions? I have Windows VM's on other hardware (typically PowerEdge R905s and R710s and they seem ok).
This 2950 is a decent machine.. 32G RAM, 2x Xeon E5405 (quad core) at 2ghz. And I'm only allocating 1 CPU and 768MB RAM for each VM.
Sample config:
name = "win01n" maxmem = 768 memory = 768 vcpus = 1 builder = "hvm" kernel = "/usr/lib/xen/boot/hvmloader" boot = "c" pae = 1 acpi = 1 apic = 0 localtime = 1 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" device_model = "/usr/lib64/xen/bin/qemu-dm" usbdevice = "tablet" sdl = 0 vnc = 1 vncunused = 0 vncdisplay = "1" disk = [ 'file:/var/xen/running/win01n,ioemu:hda,w' ] vif = [ "bridge=xenbr0" ] serial = "pty"
Thanks in advance for any insight!
- Ryan
On 01/21/2010 05:18 AM, Ryan Pugatch wrote:
Hello all,
I have a PowerEdge 2950 that I am using to host Windows XP VM's under CentOS. I am experiencing poor performance in the Xen VM's. Any time I try to do something, the CPU usage (as reported by the Windows task manager) hits 100% for a few and then drops back down. This is causing the VM's to run sluggish and be quite annoying to use. Example.. I click IE.. cpu hits 100% and IE takes a bit to open. I go to a website.. 100%.. I click a link.. 100%. This is the case even if I only have one VM running.
I now have 32 VM's running on the machine and dom0's cpu usage as reported by xm top is 622%. It seems like Xen is just performing poorly while handling these Windows VMs. Any suggestions? I have Windows VM's on other hardware (typically PowerEdge R905s and R710s and they seem ok).
This 2950 is a decent machine.. 32G RAM, 2x Xeon E5405 (quad core) at 2ghz. And I'm only allocating 1 CPU and 768MB RAM for each VM.
Sample config:
name = "win01n" maxmem = 768 memory = 768 vcpus = 1 builder = "hvm" kernel = "/usr/lib/xen/boot/hvmloader" boot = "c" pae = 1 acpi = 1 apic = 0 localtime = 1 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" device_model = "/usr/lib64/xen/bin/qemu-dm" usbdevice = "tablet" sdl = 0 vnc = 1 vncunused = 0 vncdisplay = "1" disk = [ 'file:/var/xen/running/win01n,ioemu:hda,w' ] vif = [ "bridge=xenbr0" ] serial = "pty"
Thanks in advance for any insight!
Check Pasi's advices from http://lists.centos.org/pipermail/centos-virt/2009-August/001121.html , maybe they apply to you too
On Wed, Jan 20, 2010 at 10:18:06PM -0500, Ryan Pugatch wrote:
Hello all,
I have a PowerEdge 2950 that I am using to host Windows XP VM's under CentOS. I am experiencing poor performance in the Xen VM's. Any time I try to do something, the CPU usage (as reported by the Windows task manager) hits 100% for a few and then drops back down. This is causing the VM's to run sluggish and be quite annoying to use. Example.. I click IE.. cpu hits 100% and IE takes a bit to open. I go to a website.. 100%.. I click a link.. 100%. This is the case even if I only have one VM running.
I now have 32 VM's running on the machine and dom0's cpu usage as reported by xm top is 622%. It seems like Xen is just performing poorly while handling these Windows VMs. Any suggestions? I have Windows VM's on other hardware (typically PowerEdge R905s and R710s and they seem ok).
This 2950 is a decent machine.. 32G RAM, 2x Xeon E5405 (quad core) at 2ghz. And I'm only allocating 1 CPU and 768MB RAM for each VM.
Sample config:
name = "win01n" maxmem = 768 memory = 768 vcpus = 1 builder = "hvm" kernel = "/usr/lib/xen/boot/hvmloader" boot = "c" pae = 1 acpi = 1 apic = 0 localtime = 1 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" device_model = "/usr/lib64/xen/bin/qemu-dm" usbdevice = "tablet" sdl = 0 vnc = 1 vncunused = 0 vncdisplay = "1" disk = [ 'file:/var/xen/running/win01n,ioemu:hda,w' ] vif = [ "bridge=xenbr0" ] serial = "pty"
Thanks in advance for any insight!
So I take it you're not using PV-on-HVM drivers in Windows.. ie. windows is using the emulated NIC and IDE disk?
That will be slow.
You need to use PV drivers. See GPLPV drivers: http://wiki.xensource.com/xenwiki/XenWindowsGplPv/ http://www.meadowcourt.org/downloads/
0.11.0.188 is the latest binary release.
-- Pasi
So I take it you're not using PV-on-HVM drivers in Windows.. ie. windows is using the emulated NIC and IDE disk?
That will be slow.
You need to use PV drivers. See GPLPV drivers: http://wiki.xensource.com/xenwiki/XenWindowsGplPv/ http://www.meadowcourt.org/downloads/
0.11.0.188 is the latest binary release.
-- Pasi
Thanks, will try that.
Ryan
Ryan Pugatch wrote:
So I take it you're not using PV-on-HVM drivers in Windows.. ie. windows is using the emulated NIC and IDE disk?
That will be slow.
You need to use PV drivers. See GPLPV drivers: http://wiki.xensource.com/xenwiki/XenWindowsGplPv/ http://www.meadowcourt.org/downloads/
0.11.0.188 is the latest binary release.
-- Pasi
Thanks, will try that.
Ryan
The install worked beautifully except that my VM now has an ip assigned of 192.168.122.186 which I'm guessing is in relation to virbr0 which is 192.168.122.1 on the host (and subsequently has no internet connectivity). Is it possible for me to pass through to my network like before so my VM can have an ip on my local network?
The install worked beautifully except that my VM now has an ip assigned of 192.168.122.186 which I'm guessing is in relation to virbr0 which is 192.168.122.1 on the host (and subsequently has no internet connectivity). Is it possible for me to pass through to my network like before so my VM can have an ip on my local network?
Unless you specify a bridge in the xen conf file, it uses the first one it finds alphabetically, I believe. Do you use virbr0 for anything else? As an easy way of making sure xenbr0 is used, I removed the file /etc/libvirt/qemu/networks/autostart/default.xml which was auto-starting this DHCP (and NATted?) bridge, and I only was using direct access. But you can also make it explicit on the interface line, one of my HVM VMs has the following:
vif = [ 'mac=00:16:3e:0a:4c:d2, bridge=xenbr0' ]
Eric
Eric Searcy wrote:
The install worked beautifully except that my VM now has an ip assigned of 192.168.122.186 which I'm guessing is in relation to virbr0 which is 192.168.122.1 on the host (and subsequently has no internet connectivity). Is it possible for me to pass through to my network like before so my VM can have an ip on my local network?
Unless you specify a bridge in the xen conf file, it uses the first one it finds alphabetically, I believe. Do you use virbr0 for anything else? As an easy way of making sure xenbr0 is used, I removed the file /etc/libvirt/qemu/networks/autostart/default.xml which was auto-starting this DHCP (and NATted?) bridge, and I only was using direct access. But you can also make it explicit on the interface line, one of my HVM VMs has the following:
vif = [ 'mac=00:16:3e:0a:4c:d2, bridge=xenbr0' ]
Eric
That worked.. thanks Eric. However, even with the GPLPV drivers, I am seeing a lot of CPU usage when I do anything on the Windows VM. It isn't just the guest that is reporting CPU usage, though. When I watch xm top, the CPU usage jumps on that VM. Not really sure what else can be done to improve performance.
On Thu, Jan 21, 2010 at 11:37:02AM -0500, Ryan Pugatch wrote:
Eric Searcy wrote:
The install worked beautifully except that my VM now has an ip assigned of 192.168.122.186 which I'm guessing is in relation to virbr0 which is 192.168.122.1 on the host (and subsequently has no internet connectivity). Is it possible for me to pass through to my network like before so my VM can have an ip on my local network?
Unless you specify a bridge in the xen conf file, it uses the first one it finds alphabetically, I believe. Do you use virbr0 for anything else? As an easy way of making sure xenbr0 is used, I removed the file /etc/libvirt/qemu/networks/autostart/default.xml which was auto-starting this DHCP (and NATted?) bridge, and I only was using direct access. But you can also make it explicit on the interface line, one of my HVM VMs has the following:
vif = [ 'mac=00:16:3e:0a:4c:d2, bridge=xenbr0' ]
Eric
That worked.. thanks Eric. However, even with the GPLPV drivers, I am seeing a lot of CPU usage when I do anything on the Windows VM. It isn't just the guest that is reporting CPU usage, though. When I watch xm top, the CPU usage jumps on that VM. Not really sure what else can be done to improve performance.
Check these tips: http://wiki.xensource.com/xenwiki/XenBestPractices
And also this: http://wiki.xensource.com/xenwiki/XenCommonProblems
-- Pasi
On Thu, Jan 21, 2010 at 11:28:43PM +0200, Pasi Kärkkäinen wrote:
On Thu, Jan 21, 2010 at 11:37:02AM -0500, Ryan Pugatch wrote:
Eric Searcy wrote:
The install worked beautifully except that my VM now has an ip assigned of 192.168.122.186 which I'm guessing is in relation to virbr0 which is 192.168.122.1 on the host (and subsequently has no internet connectivity). Is it possible for me to pass through to my network like before so my VM can have an ip on my local network?
Unless you specify a bridge in the xen conf file, it uses the first one it finds alphabetically, I believe. Do you use virbr0 for anything else? As an easy way of making sure xenbr0 is used, I removed the file /etc/libvirt/qemu/networks/autostart/default.xml which was auto-starting this DHCP (and NATted?) bridge, and I only was using direct access. But you can also make it explicit on the interface line, one of my HVM VMs has the following:
vif = [ 'mac=00:16:3e:0a:4c:d2, bridge=xenbr0' ]
Eric
That worked.. thanks Eric. However, even with the GPLPV drivers, I am seeing a lot of CPU usage when I do anything on the Windows VM. It isn't just the guest that is reporting CPU usage, though. When I watch xm top, the CPU usage jumps on that VM. Not really sure what else can be done to improve performance.
Check these tips: http://wiki.xensource.com/xenwiki/XenBestPractices
And also this: http://wiki.xensource.com/xenwiki/XenCommonProblems
Oh, one more thing.. is it only the guest VM taking cpu time when you monitor with "xm top"?
Do you have the latest service packs installed in the guest?
Have you tried monitoring the performance from inside the guest, to figure out what takes cpu time there?
-- Pasi
Pasi Kärkkäinen wrote:
Oh, one more thing.. is it only the guest VM taking cpu time when you monitor with "xm top"?
Do you have the latest service packs installed in the guest?
Have you tried monitoring the performance from inside the guest, to figure out what takes cpu time there?
-- Pasi
The guest is running XP w/ SP3. It is not just the guest VM taking cpu time. Basically, whatever process I am doing hits 100% cpu usage in Windows. If I open IE, it will use 100% cpu for a bit while it opens. If I try to click on a website, it will do it again. Thus, it is causing a very sluggish session as it keeps pausing. It isn't just IE, it's any program I try to use. This CPU usage is reflected in xm top as both the guest's CPU% will hit 100% as will Dom-0.
I have done pretty much everything specified in the Xen Best Practices. In fact, I have dedicated 4GB to Dom-0.
Thanks,
- Ryan
On Thu, Jan 21, 2010 at 04:39:31PM -0500, Ryan Pugatch wrote:
Pasi Kärkkäinen wrote:
Oh, one more thing.. is it only the guest VM taking cpu time when you monitor with "xm top"?
Do you have the latest service packs installed in the guest?
Have you tried monitoring the performance from inside the guest, to figure out what takes cpu time there?
-- Pasi
The guest is running XP w/ SP3. It is not just the guest VM taking cpu time. Basically, whatever process I am doing hits 100% cpu usage in Windows. If I open IE, it will use 100% cpu for a bit while it opens. If I try to click on a website, it will do it again. Thus, it is causing a very sluggish session as it keeps pausing. It isn't just IE, it's any program I try to use. This CPU usage is reflected in xm top as both the guest's CPU% will hit 100% as will Dom-0.
Did you verify from Windows Task Manager you're actually using the GPLPV PV drivers/devices?
How do you use the WinXP guest? Through the VNC console, or by using RDP over the network? RDP is recommended, since it's much faster.
Does WinXP have high IOwait?
I have done pretty much everything specified in the Xen Best Practices. In fact, I have dedicated 4GB to Dom-0.
Dom0 only needs 512 MB - 1024 MB (imho) on a common machine.
-- Pasi
On Fri, Jan 22, 2010 at 11:20:10AM +0200, Pasi Kärkkäinen wrote:
On Thu, Jan 21, 2010 at 04:39:31PM -0500, Ryan Pugatch wrote:
Pasi Kärkkäinen wrote:
Oh, one more thing.. is it only the guest VM taking cpu time when you monitor with "xm top"?
Do you have the latest service packs installed in the guest?
Have you tried monitoring the performance from inside the guest, to figure out what takes cpu time there?
-- Pasi
The guest is running XP w/ SP3. It is not just the guest VM taking cpu time. Basically, whatever process I am doing hits 100% cpu usage in Windows. If I open IE, it will use 100% cpu for a bit while it opens. If I try to click on a website, it will do it again. Thus, it is causing a very sluggish session as it keeps pausing. It isn't just IE, it's any program I try to use. This CPU usage is reflected in xm top as both the guest's CPU% will hit 100% as will Dom-0.
Did you verify from Windows Task Manager you're actually using the GPLPV PV drivers/devices?
How do you use the WinXP guest? Through the VNC console, or by using RDP over the network? RDP is recommended, since it's much faster.
Does WinXP have high IOwait?
I have done pretty much everything specified in the Xen Best Practices. In fact, I have dedicated 4GB to Dom-0.
Dom0 only needs 512 MB - 1024 MB (imho) on a common machine.
Oh, one more thing.. I noticed you had apic=0 in the WinXP config. Try changing that to 1. Not sure if that helps, but worth trying.
Also, are you running latest CentOS 5.4 with updates on dom0?
If yes, you might want to try Xen 3.4.2 from http://gitco.de/repo/ to see if that makes a difference.
-- Pasi
Pasi Kärkkäinen wrote:
Oh, one more thing.. I noticed you had apic=0 in the WinXP config. Try changing that to 1. Not sure if that helps, but worth trying.
Also, are you running latest CentOS 5.4 with updates on dom0?
If yes, you might want to try Xen 3.4.2 from http://gitco.de/repo/ to see if that makes a difference.
-- Pasi
Hi Pasi,
To answer all of your questions:
I am for sure using the GPLPV drivers.
I have used the guest via VNC and RDP and received the same performance.
The XP guest does not appear to have high i/owait.
I have tried both apic=0 and apic=1.
And my host OS is all up to date.
Xen 3.4.2 may be an option, but I think I am going to pursue VMWare ESXi at this point.
I appreciate your help and all of your suggestions.
Thanks, Ryan
On 01/22/2010 09:20 AM, Ryan Pugatch wrote:
Xen 3.4.2 may be an option, but I think I am going to pursue VMWare ESXi at this point.
Be careful with glibc versions with VMWare on CentOS. They haven't bothered to update for 5.4, and it's very crashy. You can get the libc.so from 5.3 and put it in its library path to help. Search on 'vmware glibc 5.4'.
If you're willing to switch from Xen, you might also try out KVM. It's much easier to deal with than the VMWare tomcat/browser/browser-plugin monstrosity, especially if you use X. I've got a Win2K3 machine here in an old version of KVM (less optimized) and launching Firefox only briefly moved the CPU meter to 23%. IOW, it seems normal.
-Bill
On Fri, Jan 22, 2010 at 02:27:05PM -0500, Bill McGonigle wrote:
On 01/22/2010 09:20 AM, Ryan Pugatch wrote:
Xen 3.4.2 may be an option, but I think I am going to pursue VMWare ESXi at this point.
Be careful with glibc versions with VMWare on CentOS. They haven't bothered to update for 5.4, and it's very crashy. You can get the libc.so from 5.3 and put it in its library path to help. Search on 'vmware glibc 5.4'.
If you're willing to switch from Xen, you might also try out KVM. It's much easier to deal with than the VMWare tomcat/browser/browser-plugin monstrosity, especially if you use X. I've got a Win2K3 machine here in an old version of KVM (less optimized) and launching Firefox only briefly moved the CPU meter to 23%. IOW, it seems normal.
Also many people are running Windows 2003 VMs on Xen without problems.
-- Pasi
Pasi Kärkkäinen wrote:
Also many people are running Windows 2003 VMs on Xen without problems.
-- Pasi
My patch management and antivirus console machine is a Xen virtualized Windows 2008 machine that doesn't have problems. I even moved an XP VM over to the same server as that machine just to see what happens. The XP machine still performed poorly.
Ryan
On Sun, Jan 24, 2010 at 09:03:58PM -0500, Ryan Pugatch wrote:
Pasi Kärkkäinen wrote:
Also many people are running Windows 2003 VMs on Xen without problems.
-- Pasi
My patch management and antivirus console machine is a Xen virtualized Windows 2008 machine that doesn't have problems. I even moved an XP VM over to the same server as that machine just to see what happens. The XP machine still performed poorly.
Ok. The next thing to try would be to update to gitco Xen 3.4.2 rpms, but I guess you switched to ESXi already :)
-- Pasi
On 01/25/2010 07:28 PM, Pasi Kärkkäinen wrote:
On Sun, Jan 24, 2010 at 09:03:58PM -0500, Ryan Pugatch wrote:
Pasi Kärkkäinen wrote:
Also many people are running Windows 2003 VMs on Xen without problems.
-- Pasi
My patch management and antivirus console machine is a Xen virtualized Windows 2008 machine that doesn't have problems. I even moved an XP VM over to the same server as that machine just to see what happens. The XP machine still performed poorly.
Ok. The next thing to try would be to update to gitco Xen 3.4.2 rpms, but I guess you switched to ESXi already :)
For what is worth, I had several problems with the gitco packages: - in xen-3.3 .0 , Java starting to consume all the processing power ( the same VM with stock xen from Centos 5.2 was fine in this aspect; see my thread started at http://lists.centos.org/pipermail/centos-virt/2009-February/000836.html ) - with xen-3.4.1 and 3.4.2 from gitco, an XP HVM failed to start, despite working perfectly with stock Centos 5.4
On Mon, Jan 25, 2010 at 09:24:41PM +0200, Manuel Wolfshant wrote:
On 01/25/2010 07:28 PM, Pasi Kärkkäinen wrote:
On Sun, Jan 24, 2010 at 09:03:58PM -0500, Ryan Pugatch wrote:
Pasi Kärkkäinen wrote:
Also many people are running Windows 2003 VMs on Xen without problems.
-- Pasi
My patch management and antivirus console machine is a Xen virtualized Windows 2008 machine that doesn't have problems. I even moved an XP VM over to the same server as that machine just to see what happens. The XP machine still performed poorly.
Ok. The next thing to try would be to update to gitco Xen 3.4.2 rpms, but I guess you switched to ESXi already :)
For what is worth, I had several problems with the gitco packages:
- in xen-3.3 .0 , Java starting to consume all the processing power (
the same VM with stock xen from Centos 5.2 was fine in this aspect; see my thread started at http://lists.centos.org/pipermail/centos-virt/2009-February/000836.html )
- with xen-3.4.1 and 3.4.2 from gitco, an XP HVM failed to start,
despite working perfectly with stock Centos 5.4
Ok. Did you check "xm log" for more information about the XP problem?
-- Pasi
Pasi Kärkkäinen wrote:
On Mon, Jan 25, 2010 at 09:24:41PM +0200, Manuel Wolfshant wrote:
On 01/25/2010 07:28 PM, Pasi Kärkkäinen wrote:
On Sun, Jan 24, 2010 at 09:03:58PM -0500, Ryan Pugatch wrote:
Pasi Kärkkäinen wrote:
Also many people are running Windows 2003 VMs on Xen without problems.
-- Pasi
My patch management and antivirus console machine is a Xen virtualized Windows 2008 machine that doesn't have problems. I even moved an XP VM over to the same server as that machine just to see what happens. The XP machine still performed poorly.
Ok. The next thing to try would be to update to gitco Xen 3.4.2 rpms, but I guess you switched to ESXi already :)
For what is worth, I had several problems with the gitco packages:
- in xen-3.3 .0 , Java starting to consume all the processing power (
the same VM with stock xen from Centos 5.2 was fine in this aspect; see my thread started at http://lists.centos.org/pipermail/centos-virt/2009-February/000836.html )
- with xen-3.4.1 and 3.4.2 from gitco, an XP HVM failed to start,
despite working perfectly with stock Centos 5.4
Ok. Did you check "xm log" for more information about the XP problem?
I apologize, my wording could have bean clearer: it was Windows that was crashing with the new version of xen (some "hardware" changes in the VM, I presume); xen itself was pretty happy. At the time I needed a functional XP (even if it was an experimental installation and I could do some experiments). I did not do any digging because my life as Windows programmer/admin ended exactly 10 years ago and my Windows skills were not really upgraded ever since; therefore I simply reverted to the known good configuration. That is, Centos 5.4. These days I could do some debugging if that would be useful (the XP VM is mostly useless now, it has served its purpose), but I would need a tutor to keep my hand on the Win part ( I am not willing to spend time googling for solutions to Windows issues )
The install worked beautifully except that my VM now has an
ip assigned
of 192.168.122.186 which I'm guessing is in relation to
virbr0 which is
192.168.122.1 on the host (and subsequently has no internet
connectivity). Is it possible for me to pass through to my
network like
before so my VM can have an ip on my local network?
That's the default virtual network that's installed on every host. It's useful for file transfers, etc. between guests because it transfers across the host's buses, rather than across your network switch. I find I have to create a bridge that's associated with a real/physical network card that can be shared to the vms.
If you have only one network card in the server, you should add more. 32 computers (vms) sharing one card is only one bottleneck your system has. Another would be using a file to run the vm from, rather than a block device.
Do me a favor - download the program HD Tach into one of your XP machines, and when all the other vms are idle, or turned off, run it and tell me what kind of read/write speeds you get for the vm's C: drive. I get 200 megs a second from my 4TB block device, which is 4 sata drives connected to a 3Ware controller in a raid 5.
When I read that red hat was going with kvm and might do away with xen in the future, I decided to install all my vms using only kvm. And what I found was, that using the built in virtual hardware for network cards and storage, I get native speed without having to add the virtio drivers.
Xen requires a modified kernel on the host machine and drivers loaded into the vms to get native speeds. That was always a problem for me because the drivers wouldn't always load into many linux distros I use.
compdoc wrote:
If you have only one network card in the server, you should add more. 32 computers (vms) sharing one card is only one bottleneck your system has. Another would be using a file to run the vm from, rather than a block device.
Is it possible to migrate from a block device to file?
On Thu, 2010-01-21 at 12:31 -0500, Ryan Pugatch wrote:
compdoc wrote:
If you have only one network card in the server, you should add more. 32 computers (vms) sharing one card is only one bottleneck your system has. Another would be using a file to run the vm from, rather than a block device.
Is it possible to migrate from a block device to file?
Use /bin/dd to copy to a file and change the VM conf to reflect that.
Andri
-
Andri Möll -
Bill McGonigle -
compdoc -
Eric Searcy -
Manuel Wolfshant -
Pasi Kärkkäinen -
Ryan Pugatch