We are planing on moving to xl but have not done so yet.

Our xend configs are untouched so they should match what they were when the xen-* packages were installed.

We force dom0 to 1024MB of memory via grub.

On a couple servers were we've seen this issue we've had 16+ GB of memory free and the domU was requesting something small like 2GB.

-- Shaun

------ Original Message ------ From: "George Dunlap" dunlapg@umich.edu To: "Shaun Reitan" shaun.reitan@ndchost.com; "Discussion about the virtualization on CentOS" centos-virt@centos.org Sent: 6/24/2016 4:28:53 AM Subject: Re: [CentOS-virt] Cannot allocate Memory

On Wed, Jun 22, 2016 at 6:45 PM, Shaun Reitan shaun.reitan@ndchost.com wrote:

...

Any of you guys ever seen an issue with Xen 4.4 were xm cannot create a guest because of what looks like an issue allocating memory even though xm info shows like 5x the amount of free memory needed? We are still unfortunately still using xm... it's on my list, i know..

We've had this happen on a couple hosts now. Only way to resolve seams to be rebooting the host. I'm going to update the host to latest Xen 4.4 now hoping this is a old bug.

xend hasn't had much love in years, so it's fairly unlikely that this has been fixed.

...

Here's from xen logs

[2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:105) XendDomainInfo.create(['vm', ['name', 'xxx'], ['memory', 2048], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 2], ['oos', 1], ['image', ['linux', ['kernel', '/kernels/vmlinux-2.6.18.8-4'], ['videoram', 4], ['args', 'root=/dev/xvda ro xencons=tty console=tty1 '], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['vbd', ['uname', 'phy:vg/fs_6818'], ['dev', 'xvda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:vg/fs_6819'], ['dev', 'xvdb'], ['mode', 'w']]], ['device', ['vif', ['rate', '40mb/s'], ['mac', 'FE:FD:48:01:F1:E7']]]]) [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:2504) XendDomainInfo.constructDomain [2016-06-22 09:13:50 1958] DEBUG (balloon:187) Balloon: 7602632 KiB free; need 16384; done. [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:2566) (12, 'Cannot allocate memory') Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2561, in _constructDomain target = self.info.target()) Error: (12, 'Cannot allocate memory') [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:490) VM start failed Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 475, in start XendTask.log_progress(0, 30, self._constructDomain) File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2572, in _constructDomain raise VmError(failmsg) VmError: Creating domain failed: name=xxx [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:110) Domain construction failed Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 108, in create vm.start() File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 475, in start XendTask.log_progress(0, 30, self._constructDomain) File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2572, in _constructDomain raise VmError(failmsg) VmError: Creating domain failed: name=xxx

If you haven't decided to switch to xl, would you mind reposting this question to xen-users? If you do, please also include the output of "xl info" after the failure.

The logs tell us that xend is asking dom0 to free up some memory to use to create the guest. My guess is that there's a slight mismatch between how much memory xend things needs to be freed and how much memory actually needs freeing.

Thanks, -George

We're using the stock configuration, the dom0 is limited to 1024MB via grub.

------ Original Message ------ From: "Pasi Kärkkäinen" pasik@iki.fi To: "Shaun Reitan" shaun.reitan@ndchost.com; "Discussion about the virtualization on CentOS" centos-virt@centos.org Sent: 6/26/2016 9:30:57 AM Subject: Re: [CentOS-virt] Cannot allocate Memory

On Wed, Jun 22, 2016 at 05:45:05PM +0000, Shaun Reitan wrote:

...

Any of you guys ever seen an issue with Xen 4.4 were xm cannot 

create a guest because of what looks like an issue allocating memory even though xm info shows like 5x the amount of free memory needed? We are still unfortunately still using xm... it's on my list, i know.. We've had this happen on a couple hosts now. Only way to resolve seams to be rebooting the host. I'm going to update the host to latest Xen 4.4 now hoping this is a old bug. Here's from xen logs [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:105) XendDomainInfo.create(['vm', ['name', 'xxx'], ['memory', 2048], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['vcpus', 2], ['oos', 1], ['image', ['linux', ['kernel', '/kernels/vmlinux-2.6.18.8-4'], ['videoram', 4], ['args', 'root=/dev/xvda ro xencons=tty console=tty1 '], ['tsc_mode', 0], ['nomigrate', 0]]], ['s3_integrity', 1], ['device', ['vbd', ['uname', 'phy:vg/fs_6818'], ['dev', 'xvda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'phy:vg/fs_6819'], ['dev', 'xvdb'], ['mode', 'w']]], ['device', ['vif', ['rate', '40mb/s'], ['mac', 'FE:FD:48:01:F1:E7']]]]) [2016-06-22 09:13:50 1958] DEBUG (XendDomainInfo:2504) XendDomainInfo.constructDomain [2016-06-22 09:13:50 1958] DEBUG (balloon:187) Balloon: 7602632 KiB free; need 16384; done.

Hmm.. so you're using (dom0) memory ballooning? You might want to disable it. Things worked better without ballooning (at least with xend).

-- Pasi

...

[2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:2566) (12, 

'Cannot allocate memory') Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2561, in _constructDomain target = self.info.target()) Error: (12, 'Cannot allocate memory') [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:490) VM start failed Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 475, in start XendTask.log_progress(0, 30, self._constructDomain) File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2572, in _constructDomain raise VmError(failmsg) VmError: Creating domain failed: name=xxx [2016-06-22 09:13:50 1958] ERROR (XendDomainInfo:110) Domain construction failed Traceback (most recent call last): File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 108, in create vm.start() File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 475, in start XendTask.log_progress(0, 30, self._constructDomain) File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, in log_progress retval = func(*args, **kwds) File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 2572, in _constructDomain raise VmError(failmsg) VmError: Creating domain failed: name=xxx -- Shaun