Hi All
Back in August there was a report of these errors popping up in dmesg, and as far as I can tell it's still unresolved even in the latest kernel.
System RAM resource [mem 0xc8000000-0xcfffffff] cannot be added xen_balloon: reserve_additional_memory: add_memory() failed: -17
On this example its running 3.10.20-11.el6.centos.alt.x86_64
To replicate this just disable Dom0 memory ballooning in /etc/xen/xend-config.sxp and these errors will over time completely fill up dmesg.
This is because the amount of memory within these two sys files is different:
/sys/devices/system/xen_memory/xen_memory0/info/current_kb
/sys/devices/system/xen_memory/xen_memory0/target_kb
We can manually edit to work around the problem and stop the error but its not ideal, I would help with making a patch but don't really know where to start!
Thanks Chris
On Fri, 3 Jan 2014, Chris Elliott wrote:
Hi All
Back in August there was a report of these errors popping up in dmesg, and as far as I can tell it’s still unresolved even in the latest kernel.
System RAM resource [mem 0xc8000000-0xcfffffff] cannot be added
xen_balloon: reserve_additional_memory: add_memory() failed: -17
On this example its running 3.10.20-11.el6.centos.alt.x86_64
To replicate this just disable Dom0 memory ballooning in /etc/xen/xend-config.sxp and these errors will over time completely fill up dmesg.
When you disable dom0 memory ballooning, do you also reboot passing dom0_mem=something to the Xen command line (for example dom0_mem=4G)? If you do so, do you still see those errors?
This is because the amount of memory within these two sys files is different:
/sys/devices/system/xen_memory/xen_memory0/info/current_kb
/sys/devices/system/xen_memory/xen_memory0/target_kb
We can manually edit to work around the problem and stop the error but its not ideal, I would help with making a patch but don’t really know where to start!
On 03/01/14 19:58, Chris Elliott wrote:
Hi All
Back in August there was a report of these errors popping up in dmesg, and as far as I can tell it’s still unresolved even in the latest kernel.
System RAM resource [mem 0xc8000000-0xcfffffff] cannot be added xen_balloon: reserve_additional_memory: add_memory() failed: -17
Hmmm. I definitely thought this was fixed. Can you provide your Xen command line and the kernel dmesg after boot?
David
On 03/01/14 19:58, Chris Elliott wrote:
Hi All
Back in August there was a report of these errors popping up in dmesg, and as far as I can tell it’s still unresolved even in the latest kernel.
System RAM resource [mem 0xc8000000-0xcfffffff] cannot be added xen_balloon: reserve_additional_memory: add_memory() failed: -17
Drivers (e.g., blkback) use ballooned pages for use for foreign mappings. Depending on dom0_mem options there may be no ballooned memory available at boot and dom0's balloon is increased by allocating pages. This decreases current_kb.
The balloon driver then tries to decrease the balloon to match the target but it can only do this by hotplugging new memory. A bug in the balloon driver means the location of this memory overlaps with an address range that is already in use and the hotplug will fail.
As a workaround you can boot dom0 with a suitable amount of pre-ballooned memory with:
dom0_mem=3G,max:4G
You may need to experiment with the max: value.
David
At the moment we are using dom0_mem=3G,max:3G for example (where dom0 mem and max values are the same). So we just need to allow a slightly higher max value to leave some ballooned pages?
The Xen4centos documentation also says to use the same value for dom0_mem and max.
Thanks Chris
_____ ________________________________________ From: David Vrabel [david.vrabel@citrix.com] Sent: 08 January 2014 13:12 To: Discussion about the virtualization on CentOS Cc: Chris Elliott Subject: Re: [CentOS-virt] xen_balloon: reserve_additional_memory: add_memory() Errors
On 03/01/14 19:58, Chris Elliott wrote:
Hi All
Back in August there was a report of these errors popping up in dmesg, and as far as I can tell it’s still unresolved even in the latest kernel.
System RAM resource [mem 0xc8000000-0xcfffffff] cannot be added xen_balloon: reserve_additional_memory: add_memory() failed: -17
Drivers (e.g., blkback) use ballooned pages for use for foreign mappings. Depending on dom0_mem options there may be no ballooned memory available at boot and dom0's balloon is increased by allocating pages. This decreases current_kb.
The balloon driver then tries to decrease the balloon to match the target but it can only do this by hotplugging new memory. A bug in the balloon driver means the location of this memory overlaps with an address range that is already in use and the hotplug will fail.
As a workaround you can boot dom0 with a suitable amount of pre-ballooned memory with:
dom0_mem=3G,max:4G
You may need to experiment with the max: value.
David
On 08/01/14 13:59, Chris Elliott wrote:
At the moment we are using dom0_mem=3G,max:3G for example (where dom0 mem and max values are the same). So we just need to allow a slightly higher max value to leave some ballooned pages?
Yes.
The Xen4centos documentation also says to use the same value for dom0_mem and max.
That should be updated with the workaround.
David
On 01/08/2014 08:05 AM, David Vrabel wrote:
On 08/01/14 13:59, Chris Elliott wrote:
At the moment we are using dom0_mem=3G,max:3G for example (where dom0 mem and max values are the same). So we just need to allow a slightly higher max value to leave some ballooned pages?
Yes.
The Xen4centos documentation also says to use the same value for dom0_mem and max.
That should be updated with the workaround.
David
For completeness on this issue, here are the current kernel settings for all XEN kernel variables:
CONFIG_XEN=y CONFIG_XEN_DOM0=y CONFIG_XEN_PRIVILEGED_GUEST=y CONFIG_XEN_PVHVM=y CONFIG_XEN_MAX_DOMAIN_MEMORY=500 CONFIG_XEN_SAVE_RESTORE=y CONFIG_XEN_DEBUG_FS=y CONFIG_PCI_XEN=y CONFIG_XEN_PCIDEV_FRONTEND=y CONFIG_XEN_BLKDEV_FRONTEND=m CONFIG_XEN_BLKDEV_BACKEND=m CONFIG_XEN_BLKDEV_TAP=m CONFIG_NETXEN_NIC=m CONFIG_XEN_NETDEV_FRONTEND=m CONFIG_XEN_NETDEV_BACKEND=m CONFIG_INPUT_XEN_KBDDEV_FRONTEND=y CONFIG_HVC_XEN=y CONFIG_HVC_XEN_FRONTEND=y CONFIG_XEN_WDT=m CONFIG_XEN_FBDEV_FRONTEND=y CONFIG_XEN_BALLOON=y # CONFIG_XEN_SELFBALLOONING is not set CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=y CONFIG_XEN_SCRUB_PAGES=y CONFIG_XEN_DEV_EVTCHN=m CONFIG_XEN_BACKEND=y CONFIG_XENFS=m CONFIG_XEN_COMPAT_XENFS=y CONFIG_XEN_SYS_HYPERVISOR=y CONFIG_XEN_XENBUS_FRONTEND=y CONFIG_XEN_GNTDEV=m CONFIG_XEN_GRANT_DEV_ALLOC=m CONFIG_SWIOTLB_XEN=y CONFIG_XEN_TMEM=m CONFIG_XEN_PCIDEV_BACKEND=m CONFIG_XEN_PRIVCMD=m CONFIG_XEN_ACPI_PROCESSOR=m CONFIG_XEN_MCE_LOG=y CONFIG_XEN_HAVE_PVMMU=y
As can be seen, CONFIG_XEN_SELFBALLOONING is not set ... is this a problem? (That option was decided during the beginning of the Xen4CentOS process and I'm not sure if it is still valid).
If the kernel config option is still correct and if the Xen4CentOS Docs need changing then I will make that happen.
Thanks, Johnny Hughes
On 10/01/14 12:17, Johnny Hughes wrote:
On 01/08/2014 08:05 AM, David Vrabel wrote:
On 08/01/14 13:59, Chris Elliott wrote:
At the moment we are using dom0_mem=3G,max:3G for example (where dom0 mem and max values are the same). So we just need to allow a slightly higher max value to leave some ballooned pages?
Yes.
The Xen4centos documentation also says to use the same value for dom0_mem and max.
That should be updated with the workaround.
David
For completeness on this issue, here are the current kernel settings for all XEN kernel variables:
[...]
As can be seen, CONFIG_XEN_SELFBALLOONING is not set ... is this a problem? (That option was decided during the beginning of the Xen4CentOS process and I'm not sure if it is still valid).
XEN_SELFBALLOONING enables automatic adjustment of the guest's own target based on memory usage. I don't think this a good idea, especially for a dom0.
If the kernel config option is still correct and if the Xen4CentOS Docs need changing then I will make that happen.
Thanks.
I would suggest that approx. 16 MB per VM is a reasonable amount of pre-ballooned memory.
e.g., dom0_mem=3G,max:4G would allow for approx. 64 VMs.
David
-
Chris Elliott -
David Vrabel -
Johnny Hughes -
Stefano Stabellini