[CentOS-virt] xendomains not autostarting

Mon Nov 30 16:26:17 UTC 2009
Ben M. <centos at rivint.com>

I have been scratching my head on this for days. Xendomains services 
just doesn't want to start at boot it seems, so I don't get my 
auto-domU's up without "service xendomains start" and the all start.

chkconfig looks correct, I have checked xm dmesg, dmesg, turned off 
selinux and the only "clue" I have is that the xend.log startup looks 
different than a fairly similar machine and I don't quite understand 
what it might be saying. Is dom0 crashing and restarting at machine bootup?

I have only one domU in ../auto to keep this simpler, its name is 
"v22c54" and I have one other anomaly: smartd is also not starting on 
services boot up but apparently runs fine with a manual command.

=== xend.log boot up ===
[2009-11-30 08:40:53 xend 3466] INFO (SrvDaemon:283) Xend Daemon started

[2009-11-30 08:40:53 xend 3466] INFO (SrvDaemon:287) Xend changeset: 
unavailable.

[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:228) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 
19493383087L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 
'mem_kb': 1048576L, 'domid': 0, 'max_vcpu_id': 3, 'crashed': 0, 
'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'online_vcpus': 
4, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 
'blocked': 0})

[2009-11-30 08:40:53 xend.XendDomainInfo 3466] INFO (XendDomainInfo:240) 
Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.

[2009-11-30 08:40:53 xend.XendDomainInfo 3466] WARNING 
(XendDomainInfo:262) No vm path in store for existing domain 0

[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:992) Storing VM details: {'shadow_memory': '0', 'uuid': 
'00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 
'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0', 
'vcpus': '4', 'vcpu_avail': '15', 'memory': '1024', 'on_crash': 
'restart', 'maxmem': '1024'}

[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:1027) Storing domain details: {'cpu/1/availability': 
'online', 'cpu/3/availability': 'online', 'name': 'Domain-0', 
'console/limit': '4194304', 'cpu/2/availability': 'online', 'vm': 
'/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 
'cpu/0/availability': 'online', 'memory/target': '1048576'}

[2009-11-30 08:40:53 xend 3466] DEBUG (XendDomain:163) number of vcpus 
to use is 2

[2009-11-30 08:40:53 xend 3466] INFO (SrvServer:116) unix 
path=/var/lib/xend/xend-socket

[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:1249) XendDomainInfo.handleShutdownWatch

== after manual xendomains start ==

[root at localhost ~]# service xendomains start
Restoring Xen domains: v22c54.
Starting auto Xen domains: v22c54(skip)[done]              [  OK  ]

~~ xend.log cont'd from above point ~~

[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:287) XendDomainInfo.restore(['domain', ['domid', '3'], 
['uuid', 'a3199faf-edb4-42e5-bea1-01f2df77a47f'], ['vcpus', '1'], 
['vcpu_avail', '1'], ['cpu_cap', '0'], ['cpu_weight', '256.0'], 
['memory', '512'], ['shadow_memory', '0'], ['maxmem', '512'], 
['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'v22c54'], 
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 
'restart'], ['image', ['linux', ['ramdisk', 
'/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel', 
'/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro 
root=/dev/vgcentos00/root']]], ['cpus', []], ['device', ['vif', 
['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr1'], 
['mac', '00:16:36:41:76:ae']]], ['device', ['tap', ['backend', '0'], 
['dev', 'xvda:disk'], ['uname', 
'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]], ['device', 
['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'], 
['type', 'vnc'], ['vncunused', '1'], ['xauthority', 
'/root/.Xauthority'], ['keymap', 'en-us']]], ['state', '-b----'], 
['shutdown_reason', 'poweroff'], ['cpu_time', '0.008262668'], 
['online_vcpus', '1'], ['up_time', '305.694555044'], ['start_time', 
'1259414461.79'], ['store_mfn', '1875035'], ['console_mfn', '2193022']])
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:328) parseConfig: config is ['domain', ['domid', '3'], 
['uuid', 'a3199faf-edb4-42e5-bea1-01f2df77a47f'], ['vcpus', '1'], 
['vcpu_avail', '1'], ['cpu_cap', '0'], ['cpu_weight', '256.0'], 
['memory', '512'], ['shadow_memory', '0'], ['maxmem', '512'], 
['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'v22c54'], 
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 
'restart'], ['image', ['linux', ['ramdisk', 
'/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel', 
'/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro 
root=/dev/vgcentos00/root']]], ['cpus', []], ['device', ['vif', 
['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr1'], 
['mac', '00:16:36:41:76:ae']]], ['device', ['tap', ['backend', '0'], 
['dev', 'xvda:disk'], ['uname', 
'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]], ['device', 
['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'], 
['type', 'vnc'], ['vncunused', '1'], ['xauthority', 
'/root/.Xauthority'], ['keymap', 'en-us']]], ['state', '-b----'], 
['shutdown_reason', 'poweroff'], ['cpu_time', '0.008262668'], 
['online_vcpus', '1'], ['up_time', '305.694555044'], ['start_time', 
'1259414461.79'], ['store_mfn', '1875035'], ['console_mfn', '2193022']]
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:445) parseConfig: result is {'features': None, 'image': 
['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel', 
'/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro 
root=/dev/vgcentos00/root']], 'cpus': [], 'vcpu_avail': 1, 'backend': 
[], 'uuid': 'a3199faf-edb4-42e5-bea1-01f2df77a47f', 'on_reboot': 
'restart', 'cpu_weight': 256.0, 'memory': 512, 'cpu_cap': 0, 
'localtime': None, 'timer_mode': None, 'start_time': 1259414461.79, 
'on_poweroff': 'destroy', 'on_crash': 'restart', 'device': [('vif', 
['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge', 
'xenbr1'], ['mac', '00:16:36:41:76:ae']]), ('tap', ['tap', ['backend', 
'0'], ['dev', 'xvda:disk'], ['uname', 
'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]), ('vkbd', 
['vkbd', ['backend', '0']]), ('vfb', ['vfb', ['backend', '0'], ['type', 
'vnc'], ['vncunused', '1'], ['xauthority', '/root/.Xauthority'], 
['keymap', 'en-us']])], 'bootloader': '/usr/bin/pygrub', 'maxmem': 512, 
'shadow_memory': 0, 'name': 'v22c54', 'bootloader_args': None, 'vcpus': 
1, 'cpu': None}
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:1774) XendDomainInfo.construct: None
[2009-11-30 11:17:15 xend 3466] DEBUG (balloon:145) Balloon: 7021072 KiB 
free; need 4096; done.
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:992) Storing VM details: {'shadow_memory': '0', 'uuid': 
'a3199faf-edb4-42e5-bea1-01f2df77a47f', 'on_crash': 'restart', 
'on_reboot': 'restart', 'start_time': '1259414461.79', 'on_poweroff': 
'destroy', 'name': 'v22c54', 'xend/restart_count': '0', 'vcpus': '1', 
'vcpu_avail': '1', 'memory': '512', 'bootloader': '/usr/bin/pygrub', 
'image': "(linux (ramdisk /var/lib/xen/boot_ramdisk.yFE7zn) (kernel 
/var/lib/xen/boot_kernel.bnNF6O) (args 'ro 
root=/dev/vgcentos00/root'))", 'maxmem': '512'}
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: 
writing {'state': '1', 'backend-id': '0', 'backend': 
'/local/domain/0/backend/vkbd/1/0'} to /local/domain/1/device/vkbd/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: 
writing {'frontend-id': '1', 'domain': 'v22c54', 'frontend': 
'/local/domain/1/device/vkbd/0', 'state': '1', 'online': '1'} to 
/local/domain/0/backend/vkbd/1/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: 
writing {'backend-id': '0', 'mac': '00:16:36:41:76:ae', 'handle': '0', 
'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to 
/local/domain/1/device/vif/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: 
writing {'bridge': 'xenbr1', 'domain': 'v22c54', 'handle': '0', 
'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': 
'/local/domain/1/device/vif/0', 'mac': '00:16:36:41:76:ae', 'online': 
'1', 'frontend-id': '1'} to /local/domain/0/backend/vif/1/0.
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:633) Checking for duplicate for uname: 
/var/lib/xen/images/vms/v22c54 [tap:aio:/var/lib/xen/images/vms/v22c54], 
dev: xvda:disk, mode: w
[2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up 
device number for xvda:disk: [Errno 2] No such file or directory: 
'/dev/xvda:disk'
[2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up 
device number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up 
device number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: 
writing {'backend-id': '0', 'virtual-device': '51712', 'device-type': 
'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/1/51712'} 
to /local/domain/1/device/vbd/51712.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: 
writing {'domain': 'v22c54', 'frontend': 
'/local/domain/1/device/vbd/51712', 'format': 'raw', 'dev': 'xvda', 
'state': '1', 'params': 'aio:/var/lib/xen/images/vms/v22c54', 'mode': 
'w', 'online': '1', 'frontend-id': '1', 'type': 'tap'} to 
/local/domain/0/backend/tap/1/51712.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: 
writing {'state': '1', 'backend-id': '0', 'backend': 
'/local/domain/0/backend/vfb/1/0'} to /local/domain/1/device/vfb/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: 
writing {'vncunused': '1', 'domain': 'v22c54', 'frontend': 
'/local/domain/1/device/vfb/0', 'xauthority': '/root/.Xauthority', 
'state': '1', 'keymap': 'en-us', 'online': '1', 'frontend-id': '1', 
'type': 'vnc'} to /local/domain/0/backend/vfb/1/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (vfbif:70) No VNC passwd 
configured for vfb access
[2009-11-30 11:17:15 xend 3466] DEBUG (vfbif:11) Spawn: 
['/usr/lib64/xen/bin/qemu-dm', '-M', 'xenpv', '-d', '1', '-domain-name', 
'v22c54', '-vnc', '127.0.0.1:0', '-vncunused', '-k', 'en-us']
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:1027) Storing domain details: {'console/port': '2', 
'name': 'v22c54', 'console/limit': '4194304', 'vm': 
'/vm/a3199faf-edb4-42e5-bea1-01f2df77a47f', 'domid': '1', 
'cpu/0/availability': 'online', 'memory/target': '524288', 'store/port': 
'1'}
[2009-11-30 11:17:15 xend 3466] DEBUG (XendCheckpoint:200) 
restore:shadow=0x0, _static_max=0x200, _static_min=0x200,
[2009-11-30 11:17:15 xend 3466] DEBUG (balloon:145) Balloon: 7021064 KiB 
free; need 524288; done.
[2009-11-30 11:17:15 xend 3466] DEBUG (XendCheckpoint:217) [xc_restore]: 
/usr/lib64/xen/bin/xc_restore 20 1 1 2 0 0 0
[2009-11-30 11:17:15 xend 3466] INFO (XendCheckpoint:353) 
xc_domain_restore start: p2m_size = 20800
[2009-11-30 11:17:15 xend 3466] INFO (XendCheckpoint:353) Reloading 
memory pages:   0%
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Received all 
pages (0 races)
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:3100%
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Memory 
reloaded (35912 pages)
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Domain ready 
to be built.
[2009-11-30 11:17:21 xend 3466] DEBUG (XendCheckpoint:324) store-mfn 1842329
[2009-11-30 11:17:21 xend 3466] DEBUG (XendCheckpoint:324) console-mfn 
2117145
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Restore exit 
with rc=0
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:927) XendDomainInfo.completeRestore
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:1027) Storing domain details: {'console/ring-ref': 
'2117145', 'console/port': '2', 'name': 'v22c54', 'console/limit': 
'4194304', 'vm': '/vm/a3199faf-edb4-42e5-bea1-01f2df77a47f', 'domid': 
'1', 'cpu/0/availability': 'online', 'memory/target': '524288', 
'store/ring-ref': '1842329', 'store/port': '1'}
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:943) XendDomainInfo.completeRestore done
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices vif.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0.
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG 
(XendDomainInfo:1249) XendDomainInfo.handleShutdownWatch
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) 
hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) 
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices usb.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices vbd.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices irq.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices vkbd.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) 
hotplugStatusCallback /local/domain/0/backend/vkbd/1/0/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) 
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices vfb.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) 
hotplugStatusCallback /local/domain/0/backend/vfb/1/0/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) 
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices pci.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices ioports.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices tap.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 51712.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) 
hotplugStatusCallback /local/domain/0/backend/tap/1/51712/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) 
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for 
devices vtpm.