I have a machine with 8 CPU's (2 quad xeons). (Centos 5.4 currently).
Currently, I am running 2 domUs. One with 4 CPUs and another with 2. 2 remain unused. I am using routed networking.
On the one machine with 4 CPU's, there is a cronjob at 4am which utilizes the CPU/disk pretty heavily. It is rebuilds a mailing list archive (using MHonarc) and rebuilds a search engine index (swish-e). I dont think it uses more than 1 CPU though because the index rebuilding is not multi threaded, as far as I know.
This cron runs at 4am each day and takes about 7 minutes. My ISP is telling me that their alarms are going off because they cannot ping my server between 4am and 4:07am.
It sounds like the domU is impacting the overall Xen performance. Is there anything I can do to tune this? It kinda defeats the whole Xen virtualization concept if a single CPU messes up the network. I am going to try to add a 'nice' to the crontab for the 4am job, but still, something doesn't seem correct. Any suggestions would be much appreciated. Thanks!
----- "Ken Bass" kbass@kenbass.com wrote:
It sounds like the domU is impacting the overall Xen performance. Is there anything I can do to tune this? It kinda defeats the whole Xen virtualization concept if a single CPU messes up the network. I am going to try to add a 'nice' to the crontab for the 4am job, but still, something doesn't seem correct. Any suggestions would be much appreciated. Thanks!
In high I/O environments or ones with a lot of unpredictable guests, it's a good idea to pin dom0's CPU(s) to physical cores and exclude those cores from the guests. I find that dom0 usually only needs one CPU (pinned to one core) in almost every environment. For example, on an 8 core box, dom0 gets CPU 0 (dom0-cpus 1) and all of the guests use 1-7. The occasional firewall/load balancer/something else high-I/O guest would also gets its own core, also excluded from use by the other guests, but you should avoid that until you find that it's necessary.
Not trying to take this off-topic, I think it is relevant.
Christopher: When I booted up with dom0-cpus 1, Windows 2008 could not "find" its second cpu. However, once up, I can "xm or virt-manager" it down to 1 cpu on cup 0 and everything is fine.
Is this a unique issue or do you have a tweak or hack to force smp with dom0 on one cpu?
Christopher G. Stach II wrote:
----- "Ken Bass" kbass@kenbass.com wrote:
It sounds like the domU is impacting the overall Xen performance. Is there anything I can do to tune this? It kinda defeats the whole Xen virtualization concept if a single CPU messes up the network. I am going to try to add a 'nice' to the crontab for the 4am job, but still, something doesn't seem correct. Any suggestions would be much appreciated. Thanks!
In high I/O environments or ones with a lot of unpredictable guests, it's a good idea to pin dom0's CPU(s) to physical cores and exclude those cores from the guests. I find that dom0 usually only needs one CPU (pinned to one core) in almost every environment. For example, on an 8 core box, dom0 gets CPU 0 (dom0-cpus 1) and all of the guests use 1-7. The occasional firewall/load balancer/something else high-I/O guest would also gets its own core, also excluded from use by the other guests, but you should avoid that until you find that it's necessary.
Ben M. wrote:
Not trying to take this off-topic, I think it is relevant.
Christopher: When I booted up with dom0-cpus 1, Windows 2008 could not "find" its second cpu. However, once up, I can "xm or virt-manager" it down to 1 cpu on cup 0 and everything is fine.
Is this a unique issue or do you have a tweak or hack to force smp with dom0 on one cpu?
While I was googling I saw as mention that there was setting the dom0-cpus versus modifying the dom0 kernel argument with dom0_max_vcpus=1
It sounded like using the dom0-cpus method starts the boot process with all your CPUs and then it dynamically shinks down to the dom0-cpus level you specify when the xend task starts. The kernel argument method will limit the # of CPU's from the beginning. Not sure if this makes sense in your case since you are talking about Windows 2008.
I also went into each of my guest configs and added a 'cpus="x-y"' line to tell the guest which CPUs it was pinned to. I made sure to exclude the one dom0 was running on.
----- "Ken Bass" kbass@kenbass.com wrote:
Ben M. wrote: I also went into each of my guest configs and added a 'cpus="x-y"' line to tell the guest which CPUs it was pinned to. I made sure to exclude the one dom0 was running on.
If you have your guest provisioning set up with scripts or you do it manually with virt-install, also add --cpuset=<pcpus> to the arguments.
Christopher G. Stach II wrote:
If you have your guest provisioning set up with scripts or you do it manually with virt-install, also add --cpuset=<pcpus> to the arguments.
I use a config file located in /etc/xen that I have symlinked in the auto directory. Its been a while since I set it up (Centos 5.2), but I didnt use any of the virt-manager stuff. My system sat for over a year before I deployed it. I believe I did everything by hand.
----- "Ben M." centos@rivint.com wrote:
Not trying to take this off-topic, I think it is relevant.
Christopher: When I booted up with dom0-cpus 1, Windows 2008 could not "find" its second cpu. However, once up, I can "xm or virt-manager" it down to 1 cpu on cup 0 and everything is fine.
Is this a unique issue or do you have a tweak or hack to force smp with dom0 on one cpu?
I haven't run across that personally and I don't have any 2008 guests at the moment to try it with. Did ``xm list'' show it as having multiple CPUs when the guest thought it only had one? Does it happen with 2003 or any other HVM guests?
Haven't loaded up win 2003 vms yet. So I don't have a good test case yet.
I haven't tried to repeat it in the last two kernel updates, in part because I didn't think it was an issue, I thought it was a matter that if you are going to do smp, you need at least two cpus booting up and coming into the dom0 to "activate" that in the kernel. Sorta like making babies, heh. I did try to assign it two vcpus and limited it to on cpu, but that had the same effect as going out on a date by yourself. No babies.
Both xm list and going into the Windows 2008 showed showed only 1 "cpu" present, available and taken. Rebooting a couple of times with 1 or more than one, convinced me to boot with more than one. Later I found I could xm vcpu-set and so forth and everything ran fine. I may script that up to run after startup. No harm in this requirement to me.
Christopher G. Stach II wrote:
----- "Ben M." centos@rivint.com wrote:
Not trying to take this off-topic, I think it is relevant.
Christopher: When I booted up with dom0-cpus 1, Windows 2008 could not "find" its second cpu. However, once up, I can "xm or virt-manager" it down to 1 cpu on cup 0 and everything is fine.
Is this a unique issue or do you have a tweak or hack to force smp with dom0 on one cpu?
I haven't run across that personally and I don't have any 2008 guests at the moment to try it with. Did ``xm list'' show it as having multiple CPUs when the guest thought it only had one? Does it happen with 2003 or any other HVM guests?
Christopher G. Stach II wrote:
In high I/O environments or ones with a lot of unpredictable guests, it's a good idea to pin dom0's CPU(s) to physical cores and exclude those cores from the guests. I find that dom0 usually only needs one CPU (pinned to one core) in almost every environment. For example, on an 8 core box, dom0 gets CPU 0 (dom0-cpus 1) and all of the guests us
Thanks alot for the suggestion. I went ahead and did just as you mentioned. I'll see if this fixes my networking issue when the guest runs its 4am cronjob.
-
Ben M. -
Christopher G. Stach II -
Ken Bass