Hi,
I got this error when starting a domU on Centos 5.1 i386, Device 0 (vif) could not be connected. Hotplug scripts not working.
Why is this happening ? I should restart the machine to see if this still happends after a fresh network bridge. But it's somehow a production server, I would like not to restart it, if possible. No domU is currently running. I could start domUs a week ago.
Here are the troubbleshooting information I have (everything seems fine),
- Centos 5.1 xen-3.0.3 and kernel 2.6.18-53
- ifconfig -a shows, eth0, lo, peth0, sit0, veth1, veth2, veth3, vif0.0, vif0.1, vif0.2, vid0.3 and xenbr0.
- brctl show, bridge name bridge id STP enabled interfaces xenbr0 8000.feffffffffff no peth0 vif0.0
- activated/deactivating iptables doesn't change anything. Note, securitylevel and SELinux are disabled. Currently deactivated, Chain INPUT (policy ACCEPT) target prot opt source destination
Chain FORWARD (policy ACCEPT) target prot opt source destination
Chain OUTPUT (policy ACCEPT) target prot opt source destination
- xm list -ll (domain (domid 0) (uuid 00000000-0000-0000-0000-000000000000) (vcpus 1) (cpu_weight 1.0) (memory 513) (shadow_memory 0) (maxmem 513) (features ) (name Domain-0) (on_poweroff destroy) (on_reboot restart) (on_crash restart) (state r-----) (shutdown_reason poweroff) (cpu_time 286092.966925) (online_vcpus 1) )
- loop_max is updated to 20 (which doesn't matter as I've got the vif issue, not vbd), kernel: loop: loaded (max 20 devices)
- /etc/udev/rules.d/xen-backend.rules exists, SUBSYSTEM=="xen-backend", KERNEL=="tap*", RUN+="/etc/xen/scripts/blktap $env{ACTION}" SUBSYSTEM=="xen-backend", KERNEL=="vbd*", RUN+="/etc/xen/scripts/block $env{ACTION}" SUBSYSTEM=="xen-backend", KERNEL=="vtpm*", RUN+="/etc/xen/scripts/vtpm $env{ACTION}" SUBSYSTEM=="xen-backend", KERNEL=="vif*", ACTION=="online", RUN+="$env{script} online" SUBSYSTEM=="xen-backend", KERNEL=="vif*", ACTION=="offline", RUN+="$env{script} offline" SUBSYSTEM=="xen-backend", ACTION=="remove", RUN+="/etc/xen/scripts/xen-hotplug-cleanup" KERNEL=="evtchn", NAME="xen/%k" KERNEL=="blktap[0-9]*", NAME="xen/%k"
- udev is running, root 21499 0.0 0.1 2240 624 ? S<s 15:40 0:01 /sbin/udevd -d root 22744 0.0 0.1 2044 568 pts/8 S 15:43 0:00 udevmonitor
- hald is running, 68 23539 0.0 0.6 5184 3280 ? Ss 16:23 0:00 hald root 23540 0.0 0.1 3128 952 ? S 16:23 0:00 hald-runner 68 23546 0.0 0.1 1988 776 ? S 16:23 0:00 hald-addon-keyboard: listening on /dev/input/event0
- /proc/sys/kernel/hotplug is empty (which is ok for Centos, I guess, as hald is used instead of hotplug)
- udevmonitor doesn't say anything
- /bin/sh correctly links to bash (unlike for the debian folks)
- xend.log, [2008-04-14 16:23:19 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name', 'xmb'], ['memory', '256'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['image', ['linux', ['kernel', '/data/xmb/vmlinuz'], ['ramdisk', '/data/xmb/initrd'], ['root', '/dev/hda1'], ['args', 'init=/bin/sh']]], ['device', ['vbd', ['uname', 'file:/data/xmb/xmb.ext3'], ['dev', 'hda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/data/xmb/xmb.swap'], ['dev', 'hda2'], ['mode', 'w']]]]) [2008-04-14 16:23:19 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:306) parseConfig: config is ['vm', ['name', 'xmb'], ['memory', '256'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['image', ['linux', ['kernel', '/data/xmb/vmlinuz'], ['ramdisk', '/data/xmb/initrd'], ['root', '/dev/hda1'], ['args', 'init=/bin/sh']]], ['device', ['vbd', ['uname', 'file:/data/xmb/xmb.ext3'], ['dev', 'hda1'], ['mode', 'w']]], ['device', ['vbd', ['uname', 'file:/data/xmb/xmb.swap'], ['dev', 'hda2'], ['mode', 'w']]]] [2008-04-14 16:23:19 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None, 'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None, 'image': ['linux', ['kernel', '/data/xmb/vmlinuz'], ['ramdisk', '/data/xmb/initrd'], ['root', '/dev/hda1'], ['args', 'init=/bin/sh']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'xmb', 'backend': [], 'vcpus': 1, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 256, 'device': [('vbd', ['vbd', ['uname', 'file:/data/xmb/xmb.ext3'], ['dev', 'hda1'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 'file:/data/xmb/xmb.swap'], ['dev', 'hda2'], ['mode', 'w']])], 'bootloader': None, 'cpu': None, 'maxmem': None} [2008-04-14 16:23:19 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:1349) XendDomainInfo.construct: None [2008-04-14 16:23:19 xend 14992] DEBUG (balloon:127) Balloon: 487248 KiB free; need 2048; done. [2008-04-14 16:23:19 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:1397) XendDomainInfo.initDomain: 51 1.0 [2008-04-14 16:23:19 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:1436) _initDomain:shadow_memory=0x0, maxmem=0x100, memory=0x100. [2008-04-14 16:23:19 xend 14992] DEBUG (balloon:127) Balloon: 487248 KiB free; need 262144; done. [2008-04-14 16:23:19 xend 14992] INFO (image:136) buildDomain os=linux dom=51 vcpus=1 [2008-04-14 16:23:19 xend 14992] DEBUG (image:201) domid = 51 [2008-04-14 16:23:19 xend 14992] DEBUG (image:202) memsize = 256 [2008-04-14 16:23:19 xend 14992] DEBUG (image:203) image = /data/xmb/vmlinuz [2008-04-14 16:23:19 xend 14992] DEBUG (image:204) store_evtchn = 1 [2008-04-14 16:23:19 xend 14992] DEBUG (image:205) console_evtchn = 2 [2008-04-14 16:23:19 xend 14992] DEBUG (image:206) cmdline = root=/dev/hda1 init=/bin/sh [2008-04-14 16:23:19 xend 14992] DEBUG (image:207) ramdisk = /data/xmb/initrd [2008-04-14 16:23:19 xend 14992] DEBUG (image:208) vcpus = 1 [2008-04-14 16:23:19 xend 14992] DEBUG (image:209) features = [2008-04-14 16:23:20 xend 14992] DEBUG (blkif:24) exception looking up device number for hda1: [Errno 2] No such file or directory: '/dev/hda1' [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:110) DevController: writing {'virtual-device': '769', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/51/769'} to /local/domain/51/device/vbd/769. [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:112) DevController: writing {'domain': 'xmb', 'frontend': '/local/domain/51/device/vbd/769', 'dev': 'hda1', 'state': '1', 'params': '/data/xmb/xmb.ext3', 'mode': 'w', 'online': '1', 'frontend-id': '51', 'type': 'file'} to /local/domain/0/backend/vbd/51/769. [2008-04-14 16:23:20 xend 14992] DEBUG (blkif:24) exception looking up device number for hda2: [Errno 2] No such file or directory: '/dev/hda2' [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:110) DevController: writing {'virtual-device': '770', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/51/770'} to /local/domain/51/device/vbd/770. [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:112) DevController: writing {'domain': 'xmb', 'frontend': '/local/domain/51/device/vbd/770', 'dev': 'hda2', 'state': '1', 'params': '/data/xmb/xmb.swap', 'mode': 'w', 'online': '1', 'frontend-id': '51', 'type': 'file'} to /local/domain/0/backend/vbd/51/770. [2008-04-14 16:23:20 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:756) Storing VM details: {'shadow_memory': '0', 'uuid': '196487de-140f-ef22-4b77-131313b013e8', 'on_reboot': 'restart', 'start_time': '1208183000.15', 'on_poweroff': 'destroy', 'name': 'xmb', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '256', 'on_crash': 'restart', 'image': "(linux (kernel /data/xmb/vmlinuz) (ramdisk /data/xmb/initrd) (root /dev/hda1) (args 'init=/bin/sh'))", 'maxmem': '256'} [2008-04-14 16:23:20 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:791) Storing domain details: {'console/ring-ref': '145356', 'console/port': '2', 'name': 'xmb', 'console/limit': '1048576', 'vm': '/vm/196487de-140f-ef22-4b77-131313b013e8', 'domid': '51', 'cpu/0/availability': 'online', 'memory/target': '262144', 'store/ring-ref': '145357', 'store/port': '1'} [2008-04-14 16:23:20 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:988) XendDomainInfo.handleShutdownWatch [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:143) Waiting for devices vif. [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:143) Waiting for devices usb. [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:143) Waiting for devices vbd. [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:149) Waiting for 769. [2008-04-14 16:23:20 xend 14992] DEBUG (DevController:476) hotplugStatusCallback /local/domain/0/backend/vbd/51/769/hotplug-status. [2008-04-14 16:25:00 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:1557) XendDomainInfo.destroy: domid=51 [2008-04-14 16:25:00 xend.XendDomainInfo 14992] DEBUG (XendDomainInfo:1566) XendDomainInfo.destroyDomain(51)
Note, someone reported the bug on RHEL5 x86_64; I guess I shouldn't add a command as a CentOS user (not mentioning I doubt I could reproduce the bug), https://bugzilla.redhat.com/show_bug.cgi?id=408901
Anthing I could do to dig some more ? How to fix this ? I can't play around so much on this server, although I already restarted udev and haldaemon, no changes. (I might reboot it some day but the problem may disapear then, after a fresh reboot).
Thanks
Here's the VM's config file which I forgot, name = "xmb" memory = "256" disk = ['tap:aio:/data/xmb/xmb.ext3,xvda1,w',\ 'tap:aio:/data/xmb/xmb.swap,xvda2,w'] vif = [ 'bridge=xenbr0' ] kernel = "/data/xmb/vmlinuz" ramdisk = "/data/xmb/initrd" extra = "init=/bin/sh" root = "/dev/xvda1" vcpus=1 on_reboot = 'restart' on_crash = 'restart'
Note. - I also tryed with file:...hda... (which the last email's provided logs were about) instead of tap:aio:...xvda - I tryed with nonspars too, no changes (vif error anyways, not vbd) - Forcing vif's mac address doesn't help either. - Don't bother about the use of partitions instead of a whole virtual disk (this usually works great you know that;)).
Thanks
And here's the xend-config.sxp which I also forgot: (loglevel DEBUG) (xend-unix-server yes) (xend-unix-path /var/lib/xend/xend-socket) (xend-relocation-hosts-allow '^localhost$ ^localhost\.localdomain$') #(console-limit 1024)
(network-script 'network-bridge bridge=xenbr0') #(network-script 'network-bridge netdev=eth0 bridge=xenbr0') ==> netdev peth ou eth??? #(vif-script 'vif-bridge bridge=xenbr0') (vif-script vif-bridge)
(dom0-min-mem 192) (dom0-cpus 0) #(enable-dump no) #(vnc-listen '127.0.0.1') (vncpasswd 'I_dont_use_it')
Thanks Pierre-Philipp Braun
I digged some more, I'm comming closer to an explanation. It's not only vif which is failing. If I comment vif= in VM's config, either vbd (file:..hda) or tap (tap:aio:..xvda) fails.
udev and supposedly other programs have some issues here as I used yum --installroot=/data/vmname/loop/ to install virtual machines manually, without any installer. Some rpm's postinstall scripts have probably altered the dom0 instead of installroot's directory.
Maybe I should have used rinse (@xen-tools.org) instead of my own script (MAKEDEV -d and yum --installroot) although I think the postinstall scripts shouldn't alter the underlying system when --installroot is specified. Anyways, how to fix udev ? rpm -ivhF (and -ivhf) + udevd/haldaemon restart didn't help. Note xen-backend.rules is in place.
P.S. playing with udev made the machine unavailable during a few seconds (when dealing with the network interfaces I guess), but then continued.
-
Pierre-Philipp Braun