I have been scratching my head on this for days. Xendomains services just doesn't want to start at boot it seems, so I don't get my auto-domU's up without "service xendomains start" and the all start.
chkconfig looks correct, I have checked xm dmesg, dmesg, turned off selinux and the only "clue" I have is that the xend.log startup looks different than a fairly similar machine and I don't quite understand what it might be saying. Is dom0 crashing and restarting at machine bootup?
I have only one domU in ../auto to keep this simpler, its name is "v22c54" and I have one other anomaly: smartd is also not starting on services boot up but apparently runs fine with a manual command.
=== xend.log boot up === [2009-11-30 08:40:53 xend 3466] INFO (SrvDaemon:283) Xend Daemon started
[2009-11-30 08:40:53 xend 3466] INFO (SrvDaemon:287) Xend changeset: unavailable.
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:228) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 19493383087L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 1048576L, 'domid': 0, 'max_vcpu_id': 3, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'online_vcpus': 4, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked': 0})
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] INFO (XendDomainInfo:240) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] WARNING (XendDomainInfo:262) No vm path in store for existing domain 0
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:992) Storing VM details: {'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'memory': '1024', 'on_crash': 'restart', 'maxmem': '1024'}
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:1027) Storing domain details: {'cpu/1/availability': 'online', 'cpu/3/availability': 'online', 'name': 'Domain-0', 'console/limit': '4194304', 'cpu/2/availability': 'online', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'memory/target': '1048576'}
[2009-11-30 08:40:53 xend 3466] DEBUG (XendDomain:163) number of vcpus to use is 2
[2009-11-30 08:40:53 xend 3466] INFO (SrvServer:116) unix path=/var/lib/xend/xend-socket
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:1249) XendDomainInfo.handleShutdownWatch
== after manual xendomains start ==
[root@localhost ~]# service xendomains start Restoring Xen domains: v22c54. Starting auto Xen domains: v22c54(skip)[done] [ OK ]
~~ xend.log cont'd from above point ~~
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:287) XendDomainInfo.restore(['domain', ['domid', '3'], ['uuid', 'a3199faf-edb4-42e5-bea1-01f2df77a47f'], ['vcpus', '1'], ['vcpu_avail', '1'], ['cpu_cap', '0'], ['cpu_weight', '256.0'], ['memory', '512'], ['shadow_memory', '0'], ['maxmem', '512'], ['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'v22c54'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel', '/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro root=/dev/vgcentos00/root']]], ['cpus', []], ['device', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr1'], ['mac', '00:16:36:41:76:ae']]], ['device', ['tap', ['backend', '0'], ['dev', 'xvda:disk'], ['uname', 'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]], ['device', ['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['xauthority', '/root/.Xauthority'], ['keymap', 'en-us']]], ['state', '-b----'], ['shutdown_reason', 'poweroff'], ['cpu_time', '0.008262668'], ['online_vcpus', '1'], ['up_time', '305.694555044'], ['start_time', '1259414461.79'], ['store_mfn', '1875035'], ['console_mfn', '2193022']]) [2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:328) parseConfig: config is ['domain', ['domid', '3'], ['uuid', 'a3199faf-edb4-42e5-bea1-01f2df77a47f'], ['vcpus', '1'], ['vcpu_avail', '1'], ['cpu_cap', '0'], ['cpu_weight', '256.0'], ['memory', '512'], ['shadow_memory', '0'], ['maxmem', '512'], ['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'v22c54'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel', '/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro root=/dev/vgcentos00/root']]], ['cpus', []], ['device', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr1'], ['mac', '00:16:36:41:76:ae']]], ['device', ['tap', ['backend', '0'], ['dev', 'xvda:disk'], ['uname', 'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]], ['device', ['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['xauthority', '/root/.Xauthority'], ['keymap', 'en-us']]], ['state', '-b----'], ['shutdown_reason', 'poweroff'], ['cpu_time', '0.008262668'], ['online_vcpus', '1'], ['up_time', '305.694555044'], ['start_time', '1259414461.79'], ['store_mfn', '1875035'], ['console_mfn', '2193022']] [2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:445) parseConfig: result is {'features': None, 'image': ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel', '/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro root=/dev/vgcentos00/root']], 'cpus': [], 'vcpu_avail': 1, 'backend': [], 'uuid': 'a3199faf-edb4-42e5-bea1-01f2df77a47f', 'on_reboot': 'restart', 'cpu_weight': 256.0, 'memory': 512, 'cpu_cap': 0, 'localtime': None, 'timer_mode': None, 'start_time': 1259414461.79, 'on_poweroff': 'destroy', 'on_crash': 'restart', 'device': [('vif', ['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr1'], ['mac', '00:16:36:41:76:ae']]), ('tap', ['tap', ['backend', '0'], ['dev', 'xvda:disk'], ['uname', 'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]), ('vkbd', ['vkbd', ['backend', '0']]), ('vfb', ['vfb', ['backend', '0'], ['type', 'vnc'], ['vncunused', '1'], ['xauthority', '/root/.Xauthority'], ['keymap', 'en-us']])], 'bootloader': '/usr/bin/pygrub', 'maxmem': 512, 'shadow_memory': 0, 'name': 'v22c54', 'bootloader_args': None, 'vcpus': 1, 'cpu': None} [2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:1774) XendDomainInfo.construct: None [2009-11-30 11:17:15 xend 3466] DEBUG (balloon:145) Balloon: 7021072 KiB free; need 4096; done. [2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:992) Storing VM details: {'shadow_memory': '0', 'uuid': 'a3199faf-edb4-42e5-bea1-01f2df77a47f', 'on_crash': 'restart', 'on_reboot': 'restart', 'start_time': '1259414461.79', 'on_poweroff': 'destroy', 'name': 'v22c54', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '512', 'bootloader': '/usr/bin/pygrub', 'image': "(linux (ramdisk /var/lib/xen/boot_ramdisk.yFE7zn) (kernel /var/lib/xen/boot_kernel.bnNF6O) (args 'ro root=/dev/vgcentos00/root'))", 'maxmem': '512'} [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vkbd/1/0'} to /local/domain/1/device/vkbd/0. [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: writing {'frontend-id': '1', 'domain': 'v22c54', 'frontend': '/local/domain/1/device/vkbd/0', 'state': '1', 'online': '1'} to /local/domain/0/backend/vkbd/1/0. [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: writing {'backend-id': '0', 'mac': '00:16:36:41:76:ae', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0. [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: writing {'bridge': 'xenbr1', 'domain': 'v22c54', 'handle': '0', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/1/device/vif/0', 'mac': '00:16:36:41:76:ae', 'online': '1', 'frontend-id': '1'} to /local/domain/0/backend/vif/1/0. [2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:633) Checking for duplicate for uname: /var/lib/xen/images/vms/v22c54 [tap:aio:/var/lib/xen/images/vms/v22c54], dev: xvda:disk, mode: w [2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up device number for xvda:disk: [Errno 2] No such file or directory: '/dev/xvda:disk' [2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda' [2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda' [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: writing {'backend-id': '0', 'virtual-device': '51712', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/1/51712'} to /local/domain/1/device/vbd/51712. [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: writing {'domain': 'v22c54', 'frontend': '/local/domain/1/device/vbd/51712', 'format': 'raw', 'dev': 'xvda', 'state': '1', 'params': 'aio:/var/lib/xen/images/vms/v22c54', 'mode': 'w', 'online': '1', 'frontend-id': '1', 'type': 'tap'} to /local/domain/0/backend/tap/1/51712. [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/1/0'} to /local/domain/1/device/vfb/0. [2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController: writing {'vncunused': '1', 'domain': 'v22c54', 'frontend': '/local/domain/1/device/vfb/0', 'xauthority': '/root/.Xauthority', 'state': '1', 'keymap': 'en-us', 'online': '1', 'frontend-id': '1', 'type': 'vnc'} to /local/domain/0/backend/vfb/1/0. [2009-11-30 11:17:15 xend 3466] DEBUG (vfbif:70) No VNC passwd configured for vfb access [2009-11-30 11:17:15 xend 3466] DEBUG (vfbif:11) Spawn: ['/usr/lib64/xen/bin/qemu-dm', '-M', 'xenpv', '-d', '1', '-domain-name', 'v22c54', '-vnc', '127.0.0.1:0', '-vncunused', '-k', 'en-us'] [2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:1027) Storing domain details: {'console/port': '2', 'name': 'v22c54', 'console/limit': '4194304', 'vm': '/vm/a3199faf-edb4-42e5-bea1-01f2df77a47f', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '524288', 'store/port': '1'} [2009-11-30 11:17:15 xend 3466] DEBUG (XendCheckpoint:200) restore:shadow=0x0, _static_max=0x200, _static_min=0x200, [2009-11-30 11:17:15 xend 3466] DEBUG (balloon:145) Balloon: 7021064 KiB free; need 524288; done. [2009-11-30 11:17:15 xend 3466] DEBUG (XendCheckpoint:217) [xc_restore]: /usr/lib64/xen/bin/xc_restore 20 1 1 2 0 0 0 [2009-11-30 11:17:15 xend 3466] INFO (XendCheckpoint:353) xc_domain_restore start: p2m_size = 20800 [2009-11-30 11:17:15 xend 3466] INFO (XendCheckpoint:353) Reloading memory pages: 0% [2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Received all pages (0 races) [2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:3100% [2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Memory reloaded (35912 pages) [2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Domain ready to be built. [2009-11-30 11:17:21 xend 3466] DEBUG (XendCheckpoint:324) store-mfn 1842329 [2009-11-30 11:17:21 xend 3466] DEBUG (XendCheckpoint:324) console-mfn 2117145 [2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Restore exit with rc=0 [2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:927) XendDomainInfo.completeRestore [2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:1027) Storing domain details: {'console/ring-ref': '2117145', 'console/port': '2', 'name': 'v22c54', 'console/limit': '4194304', 'vm': '/vm/a3199faf-edb4-42e5-bea1-01f2df77a47f', 'domid': '1', 'cpu/0/availability': 'online', 'memory/target': '524288', 'store/ring-ref': '1842329', 'store/port': '1'} [2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:943) XendDomainInfo.completeRestore done [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices vif. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0. [2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG (XendDomainInfo:1249) XendDomainInfo.handleShutdownWatch [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices usb. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices vbd. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices irq. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices vkbd. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vkbd/1/0/hotplug-status. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices vfb. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vfb/1/0/hotplug-status. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices pci. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices ioports. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices tap. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 51712. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/tap/1/51712/hotplug-status. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for devices vtpm.