Hi,
This is slightly OT as xen is not officially supported in EL6. But maybe someone else is running it on centos6 as well and could help me out here:
I have problems allocating a static amount of memory for my xen dom0. It seems to reduce when new domUs are created, even though I *think* ballooning is disabled. Can someone advise me what I'm doing wrong?
I have xen-4.1.1-3 installed from http://xenbits.xen.org/people/mayoung/EL6.xen on my centos6 server. I am running 2.6.32-131.12.1.el6.xendom0.x86_64 kernel available from the same place. 'xl info' reports about available memory as follows:
total_memory : 24567 free_memory : 3698 xen_commandline : dom0_mem=2048M
and in xend-config.sxp I have:
(dom0-min-mem 256) (enable-dom0-ballooning no) (total_available_memory 0)
Now one would think that I should have 2048M memory for dom0, right? I started from there, but when starting domUs it has been reduced to 1140M even though I think ballooning is disabled and there is free memory available? And the weird thing is that dom0 has done some swapping:
# free -mt total used free shared buffers cached Mem: 1140 1052 88 0 2 12 -/+ buffers/cache: 1038 102 Swap: 8194 85 8109 Total: 9335 1138 8197
Shutting down domUs and making more memory available does not reverse the effect, dom0 does not allocate more memory.
What am I missing here or is my xen setup acting strangely here?
Best regards, Peter
On Thu, Oct 20, 2011 at 12:37:34AM +0300, Peter Peltonen wrote:
Hi,
This is slightly OT as xen is not officially supported in EL6. But maybe someone else is running it on centos6 as well and could help me out here:
I have problems allocating a static amount of memory for my xen dom0. It seems to reduce when new domUs are created, even though I *think* ballooning is disabled. Can someone advise me what I'm doing wrong?
I have xen-4.1.1-3 installed from http://xenbits.xen.org/people/mayoung/EL6.xen on my centos6 server. I am running 2.6.32-131.12.1.el6.xendom0.x86_64 kernel available from the same place. 'xl info' reports about available memory as follows:
total_memory : 24567 free_memory : 3698 xen_commandline : dom0_mem=2048M
and in xend-config.sxp I have:
(dom0-min-mem 256) (enable-dom0-ballooning no) (total_available_memory 0)
Now one would think that I should have 2048M memory for dom0, right? I started from there, but when starting domUs it has been reduced to 1140M even though I think ballooning is disabled and there is free memory available? And the weird thing is that dom0 has done some swapping:
Did you check the amount of memory in dom0 right after boot? You *might* need to add also "mem=2028M" cmdline option for dom0 linux kernel (vmlinuz).
Also Linux kernel is *supposed* to swap all inactive pages!
So it's normal to have swap being used. If you have *active* swap usage all the time (pages in and out from swap), only then it means you don't have enough memory allocated.
# free -mt total used free shared buffers cached Mem: 1140 1052 88 0 2 12 -/+ buffers/cache: 1038 102 Swap: 8194 85 8109 Total: 9335 1138 8197
Shutting down domUs and making more memory available does not reverse the effect, dom0 does not allocate more memory.
dom0 is not supposed to allocate more memory! dom0 is a virtual machine. Free memory goes into Xen hypervisor, check with "xl info" or "xm info".
What am I missing here or is my xen setup acting strangely here?
You're missing the fact that dom0 is a VM.
-- Pasi
Hi,
On Sat, Oct 22, 2011 at 4:21 PM, Pasi Kärkkäinen pasik@iki.fi wrote:
and in xend-config.sxp I have:
(dom0-min-mem 256) (enable-dom0-ballooning no) (total_available_memory 0)
Now one would think that I should have 2048M memory for dom0, right? I started from there, but when starting domUs it has been reduced to 1140M even though I think ballooning is disabled and there is free memory available? And the weird thing is that dom0 has done some swapping:
Did you check the amount of memory in dom0 right after boot? You *might* need to add also "mem=2028M" cmdline option for dom0 linux kernel (vmlinuz).
Thanks for the tip! I also googled a tip to set dom0-min-mem to the same amount. I will try this next.
Shutting down domUs and making more memory available does not reverse the effect, dom0 does not allocate more memory.
dom0 is not supposed to allocate more memory! dom0 is a virtual machine. Free memory goes into Xen hypervisor, check with "xl info" or "xm info".
I am aware that dom0 is a virtual machine. As I reported earlier, xl info shows:
total_memory : 24567 free_memory : 3698 xen_commandline : dom0_mem=2048M
There is free memory. Dom0 is supposed to have 2048M memory. Why is it I have only 1140M memory in use with dom0? When I first booted the machine I used to have 2048M memory and then Xen started reducing the memory when I started more domUs.
So my problem is: Why is Xen reducing the amount of memory in dom0, even I have disabled ballooning and there is more than enough memory available for the dom0 and domUs?
Just a moment ago I experienced the following: I shut down a domU having 4096M memory. Then I tried to start it again and Xen complained that there is no memory available. The second restart produced the following error:
libxl: error: libxl_dom.c:168:libxl__build_pv xc_dom_kernel_file failed: Bad address cannot (re-)build domain: -3
There is something fishy going around... I will now try to yum update dom0 and domUs, set the dom0-min-mem to 2048M and reboot and see what happens.
Best regards, Peter
Replying to myself:
On Sun, Oct 23, 2011 at 5:51 PM, Peter Peltonen peter.peltonen@gmail.com wrote:
There is something fishy going around... I will now try to yum update dom0 and domUs, set the dom0-min-mem to 2048M and reboot and see what happens.
After rebooting I seemed to have 2048M in dom0 regardless of how many domUs were running.
But when trying to create the last domU, I was informed that xend was out of memory. At this time I still had something about 5G of real memory available for the last domU which I have configured to consume 4G of memory.
I tweaked grub.conf and xend-config.sxp and set 4096M as the dom0 memory and rebooted again. I reduced the domU memory sizes accordingly and all domUs got created without errors. But the strange thing is again dom0 memory which has been reduced from 4096M to 2052M:
[root@si ~]# cat /etc/xen/xend-config.sxp |grep dom0-min-mem # dom0-min-mem is the lowest permissible memory level (in MB) for dom0. (dom0-min-mem 4096)
[root@si ~]# xl info |grep mem total_memory : 24567 free_memory : 5853 xen_commandline : dom0_mem=4096M dom0_max_vcpus=1 dom0_vcpus_pin
[root@si ~]# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2052 1 r----- 265.6
xl top reports "(MEM)k" as "2101272" and "MAXMEM(k)" as "no limit".
In CentOS5 with Xen3 I have been used to having the amount of memory in dom0 which I have specified. My Xen4 installation seems not to do this.
If someone else is using the Xen4 packages from http://xenbits.xen.org/people/mayoung/EL6.xen I would be interested in hearing about your experiences?
BR, Peter
On Sun, Oct 23, 2011 at 08:54:13PM +0300, Peter Peltonen wrote:
Replying to myself:
On Sun, Oct 23, 2011 at 5:51 PM, Peter Peltonen peter.peltonen@gmail.com wrote:
There is something fishy going around... I will now try to yum update dom0 and domUs, set the dom0-min-mem to 2048M and reboot and see what happens.
After rebooting I seemed to have 2048M in dom0 regardless of how many domUs were running.
But when trying to create the last domU, I was informed that xend was out of memory. At this time I still had something about 5G of real memory available for the last domU which I have configured to consume 4G of memory.
I tweaked grub.conf and xend-config.sxp and set 4096M as the dom0 memory and rebooted again. I reduced the domU memory sizes accordingly and all domUs got created without errors. But the strange thing is again dom0 memory which has been reduced from 4096M to 2052M:
[root@si ~]# cat /etc/xen/xend-config.sxp |grep dom0-min-mem # dom0-min-mem is the lowest permissible memory level (in MB) for dom0. (dom0-min-mem 4096)
[root@si ~]# xl info |grep mem total_memory : 24567 free_memory : 5853 xen_commandline : dom0_mem=4096M dom0_max_vcpus=1 dom0_vcpus_pin
[root@si ~]# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2052 1 r----- 265.6
xl top reports "(MEM)k" as "2101272" and "MAXMEM(k)" as "no limit".
In CentOS5 with Xen3 I have been used to having the amount of memory in dom0 which I have specified. My Xen4 installation seems not to do this.
If someone else is using the Xen4 packages from http://xenbits.xen.org/people/mayoung/EL6.xen I would be interested in hearing about your experiences?
Replying to an old thread.. two things:
1) you might need to use "dom0_mem=2048M,max:2048M" parameter for Xen hypervisor to give the maximum upper limit for dom0.
2) (Earlier) there was a bug in the dom0 kernel code where it allocated wrong amount of memory; you had to use "mem=2048M" option for Linux dom0 kernel in addition to the dom0_mem= option for Xen to get the correct amount of memory for dom0.
-- Pasi
-
Pasi Kärkkäinen -
Peter Peltonen