I have been scratching my head on this for days. Xendomains services
just doesn't want to start at boot it seems, so I don't get my
auto-domU's up without "service xendomains start" and the all start.
chkconfig looks correct, I have checked xm dmesg, dmesg, turned off
selinux and the only "clue" I have is that the xend.log startup looks
different than a fairly similar machine and I don't quite understand
what it might be saying. Is dom0 crashing and restarting at machine bootup?
I have only one domU in ../auto to keep this simpler, its name is
"v22c54" and I have one other anomaly: smartd is also not starting on
services boot up but apparently runs fine with a manual command.
=== xend.log boot up ===
[2009-11-30 08:40:53 xend 3466] INFO (SrvDaemon:283) Xend Daemon started
[2009-11-30 08:40:53 xend 3466] INFO (SrvDaemon:287) Xend changeset:
unavailable.
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:228) XendDomainInfo.recreate({'paused': 0, 'cpu_time':
19493383087L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0,
'mem_kb': 1048576L, 'domid': 0, 'max_vcpu_id': 3, 'crashed': 0,
'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'online_vcpus':
4, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
'blocked': 0})
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] INFO (XendDomainInfo:240)
Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000.
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] WARNING
(XendDomainInfo:262) No vm path in store for existing domain 0
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:992) Storing VM details: {'shadow_memory': '0', 'uuid':
'00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart',
'on_poweroff': 'destroy', 'name': 'Domain-0', 'xend/restart_count': '0',
'vcpus': '4', 'vcpu_avail': '15', 'memory': '1024', 'on_crash':
'restart', 'maxmem': '1024'}
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:1027) Storing domain details: {'cpu/1/availability':
'online', 'cpu/3/availability': 'online', 'name': 'Domain-0',
'console/limit': '4194304', 'cpu/2/availability': 'online', 'vm':
'/vm/00000000-0000-0000-0000-000000000000', 'domid': '0',
'cpu/0/availability': 'online', 'memory/target': '1048576'}
[2009-11-30 08:40:53 xend 3466] DEBUG (XendDomain:163) number of vcpus
to use is 2
[2009-11-30 08:40:53 xend 3466] INFO (SrvServer:116) unix
path=/var/lib/xend/xend-socket
[2009-11-30 08:40:53 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:1249) XendDomainInfo.handleShutdownWatch
== after manual xendomains start ==
[root at localhost ~]# service xendomains start
Restoring Xen domains: v22c54.
Starting auto Xen domains: v22c54(skip)[done] [ OK ]
~~ xend.log cont'd from above point ~~
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:287) XendDomainInfo.restore(['domain', ['domid', '3'],
['uuid', 'a3199faf-edb4-42e5-bea1-01f2df77a47f'], ['vcpus', '1'],
['vcpu_avail', '1'], ['cpu_cap', '0'], ['cpu_weight', '256.0'],
['memory', '512'], ['shadow_memory', '0'], ['maxmem', '512'],
['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'v22c54'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
'restart'], ['image', ['linux', ['ramdisk',
'/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel',
'/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro
root=/dev/vgcentos00/root']]], ['cpus', []], ['device', ['vif',
['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr1'],
['mac', '00:16:36:41:76:ae']]], ['device', ['tap', ['backend', '0'],
['dev', 'xvda:disk'], ['uname',
'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]], ['device',
['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'],
['type', 'vnc'], ['vncunused', '1'], ['xauthority',
'/root/.Xauthority'], ['keymap', 'en-us']]], ['state', '-b----'],
['shutdown_reason', 'poweroff'], ['cpu_time', '0.008262668'],
['online_vcpus', '1'], ['up_time', '305.694555044'], ['start_time',
'1259414461.79'], ['store_mfn', '1875035'], ['console_mfn', '2193022']])
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:328) parseConfig: config is ['domain', ['domid', '3'],
['uuid', 'a3199faf-edb4-42e5-bea1-01f2df77a47f'], ['vcpus', '1'],
['vcpu_avail', '1'], ['cpu_cap', '0'], ['cpu_weight', '256.0'],
['memory', '512'], ['shadow_memory', '0'], ['maxmem', '512'],
['bootloader', '/usr/bin/pygrub'], ['features'], ['name', 'v22c54'],
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
'restart'], ['image', ['linux', ['ramdisk',
'/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel',
'/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro
root=/dev/vgcentos00/root']]], ['cpus', []], ['device', ['vif',
['backend', '0'], ['script', 'vif-bridge'], ['bridge', 'xenbr1'],
['mac', '00:16:36:41:76:ae']]], ['device', ['tap', ['backend', '0'],
['dev', 'xvda:disk'], ['uname',
'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]], ['device',
['vkbd', ['backend', '0']]], ['device', ['vfb', ['backend', '0'],
['type', 'vnc'], ['vncunused', '1'], ['xauthority',
'/root/.Xauthority'], ['keymap', 'en-us']]], ['state', '-b----'],
['shutdown_reason', 'poweroff'], ['cpu_time', '0.008262668'],
['online_vcpus', '1'], ['up_time', '305.694555044'], ['start_time',
'1259414461.79'], ['store_mfn', '1875035'], ['console_mfn', '2193022']]
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:445) parseConfig: result is {'features': None, 'image':
['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.yFE7zn'], ['kernel',
'/var/lib/xen/boot_kernel.bnNF6O'], ['args', 'ro
root=/dev/vgcentos00/root']], 'cpus': [], 'vcpu_avail': 1, 'backend':
[], 'uuid': 'a3199faf-edb4-42e5-bea1-01f2df77a47f', 'on_reboot':
'restart', 'cpu_weight': 256.0, 'memory': 512, 'cpu_cap': 0,
'localtime': None, 'timer_mode': None, 'start_time': 1259414461.79,
'on_poweroff': 'destroy', 'on_crash': 'restart', 'device': [('vif',
['vif', ['backend', '0'], ['script', 'vif-bridge'], ['bridge',
'xenbr1'], ['mac', '00:16:36:41:76:ae']]), ('tap', ['tap', ['backend',
'0'], ['dev', 'xvda:disk'], ['uname',
'tap:aio:/var/lib/xen/images/vms/v22c54'], ['mode', 'w']]), ('vkbd',
['vkbd', ['backend', '0']]), ('vfb', ['vfb', ['backend', '0'], ['type',
'vnc'], ['vncunused', '1'], ['xauthority', '/root/.Xauthority'],
['keymap', 'en-us']])], 'bootloader': '/usr/bin/pygrub', 'maxmem': 512,
'shadow_memory': 0, 'name': 'v22c54', 'bootloader_args': None, 'vcpus':
1, 'cpu': None}
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:1774) XendDomainInfo.construct: None
[2009-11-30 11:17:15 xend 3466] DEBUG (balloon:145) Balloon: 7021072 KiB
free; need 4096; done.
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:992) Storing VM details: {'shadow_memory': '0', 'uuid':
'a3199faf-edb4-42e5-bea1-01f2df77a47f', 'on_crash': 'restart',
'on_reboot': 'restart', 'start_time': '1259414461.79', 'on_poweroff':
'destroy', 'name': 'v22c54', 'xend/restart_count': '0', 'vcpus': '1',
'vcpu_avail': '1', 'memory': '512', 'bootloader': '/usr/bin/pygrub',
'image': "(linux (ramdisk /var/lib/xen/boot_ramdisk.yFE7zn) (kernel
/var/lib/xen/boot_kernel.bnNF6O) (args 'ro
root=/dev/vgcentos00/root'))", 'maxmem': '512'}
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController:
writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vkbd/1/0'} to /local/domain/1/device/vkbd/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController:
writing {'frontend-id': '1', 'domain': 'v22c54', 'frontend':
'/local/domain/1/device/vkbd/0', 'state': '1', 'online': '1'} to
/local/domain/0/backend/vkbd/1/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController:
writing {'backend-id': '0', 'mac': '00:16:36:41:76:ae', 'handle': '0',
'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to
/local/domain/1/device/vif/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController:
writing {'bridge': 'xenbr1', 'domain': 'v22c54', 'handle': '0',
'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend':
'/local/domain/1/device/vif/0', 'mac': '00:16:36:41:76:ae', 'online':
'1', 'frontend-id': '1'} to /local/domain/0/backend/vif/1/0.
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:633) Checking for duplicate for uname:
/var/lib/xen/images/vms/v22c54 [tap:aio:/var/lib/xen/images/vms/v22c54],
dev: xvda:disk, mode: w
[2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up
device number for xvda:disk: [Errno 2] No such file or directory:
'/dev/xvda:disk'
[2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up
device number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-11-30 11:17:15 xend 3466] DEBUG (blkif:27) exception looking up
device number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController:
writing {'backend-id': '0', 'virtual-device': '51712', 'device-type':
'disk', 'state': '1', 'backend': '/local/domain/0/backend/tap/1/51712'}
to /local/domain/1/device/vbd/51712.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController:
writing {'domain': 'v22c54', 'frontend':
'/local/domain/1/device/vbd/51712', 'format': 'raw', 'dev': 'xvda',
'state': '1', 'params': 'aio:/var/lib/xen/images/vms/v22c54', 'mode':
'w', 'online': '1', 'frontend-id': '1', 'type': 'tap'} to
/local/domain/0/backend/tap/1/51712.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:114) DevController:
writing {'state': '1', 'backend-id': '0', 'backend':
'/local/domain/0/backend/vfb/1/0'} to /local/domain/1/device/vfb/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (DevController:116) DevController:
writing {'vncunused': '1', 'domain': 'v22c54', 'frontend':
'/local/domain/1/device/vfb/0', 'xauthority': '/root/.Xauthority',
'state': '1', 'keymap': 'en-us', 'online': '1', 'frontend-id': '1',
'type': 'vnc'} to /local/domain/0/backend/vfb/1/0.
[2009-11-30 11:17:15 xend 3466] DEBUG (vfbif:70) No VNC passwd
configured for vfb access
[2009-11-30 11:17:15 xend 3466] DEBUG (vfbif:11) Spawn:
['/usr/lib64/xen/bin/qemu-dm', '-M', 'xenpv', '-d', '1', '-domain-name',
'v22c54', '-vnc', '127.0.0.1:0', '-vncunused', '-k', 'en-us']
[2009-11-30 11:17:15 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:1027) Storing domain details: {'console/port': '2',
'name': 'v22c54', 'console/limit': '4194304', 'vm':
'/vm/a3199faf-edb4-42e5-bea1-01f2df77a47f', 'domid': '1',
'cpu/0/availability': 'online', 'memory/target': '524288', 'store/port':
'1'}
[2009-11-30 11:17:15 xend 3466] DEBUG (XendCheckpoint:200)
restore:shadow=0x0, _static_max=0x200, _static_min=0x200,
[2009-11-30 11:17:15 xend 3466] DEBUG (balloon:145) Balloon: 7021064 KiB
free; need 524288; done.
[2009-11-30 11:17:15 xend 3466] DEBUG (XendCheckpoint:217) [xc_restore]:
/usr/lib64/xen/bin/xc_restore 20 1 1 2 0 0 0
[2009-11-30 11:17:15 xend 3466] INFO (XendCheckpoint:353)
xc_domain_restore start: p2m_size = 20800
[2009-11-30 11:17:15 xend 3466] INFO (XendCheckpoint:353) Reloading
memory pages: 0%
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Received all
pages (0 races)
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:3100%
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Memory
reloaded (35912 pages)
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Domain ready
to be built.
[2009-11-30 11:17:21 xend 3466] DEBUG (XendCheckpoint:324) store-mfn 1842329
[2009-11-30 11:17:21 xend 3466] DEBUG (XendCheckpoint:324) console-mfn
2117145
[2009-11-30 11:17:21 xend 3466] INFO (XendCheckpoint:353) Restore exit
with rc=0
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:927) XendDomainInfo.completeRestore
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:1027) Storing domain details: {'console/ring-ref':
'2117145', 'console/port': '2', 'name': 'v22c54', 'console/limit':
'4194304', 'vm': '/vm/a3199faf-edb4-42e5-bea1-01f2df77a47f', 'domid':
'1', 'cpu/0/availability': 'online', 'memory/target': '524288',
'store/ring-ref': '1842329', 'store/port': '1'}
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:943) XendDomainInfo.completeRestore done
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices vif.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0.
[2009-11-30 11:17:21 xend.XendDomainInfo 3466] DEBUG
(XendDomainInfo:1249) XendDomainInfo.handleShutdownWatch
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices usb.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices vbd.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices irq.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices vkbd.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vkbd/1/0/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices vfb.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 0.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vfb/1/0/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices pci.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices ioports.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices tap.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:164) Waiting for 51712.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/tap/1/51712/hotplug-status.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-30 11:17:21 xend 3466] DEBUG (DevController:158) Waiting for
devices vtpm.