Hi all,
I am in a bit of a fix here, I have several Guests paravirtualized running on a Dom0 that is currently running the 2.6.18-92.1.22.el5xen Kernel. All my guests (administered by different people) have subsequently been updated to later versions of the kernel (namely 2.6.18-164.el5xen or 2.6.18-128.4.1.el5xen).
Ones that are still running are using the Kernel 2.6.18-128.4.1.el5xen while Dom0 is still booted up into 2.6.18-92.1.22.el5xen
I restarted a Guest that's now trying to boot with the 2.6.18-164.el5xen Kernel and it spits out messages like XENBUS: Waiting for devices to initialise: 295s...290s...
and eventually reboots.
If I try and interrupt the boot and choose a previous version of the Kernel on the Guest, it says the following and fails to boot. The guests use a single disk image that has the SWAP and Root partitions.
Is this happening because the version of the kernel on the Dom0 is way too old for guests to boot? I am just a bit hesitant in rebooting my server for the new kernel to take effect unless I am sure :)
device-mapper: dm-raid45: initialized v0.2429 Scanning and configuring dmraid supported devices Scanning logical volumes Reading all physical volumes. This may take a while... Activating logical volumes Volume group "VolGroup00" not found Creating root device. Mounting root filesystem. mount: could not find filesystem '/dev/root' Setting up other filesystems. Setting up new root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Kernel panic - not syncing: Attempted to kill init!
On Sun, Sep 27, 2009 at 12:46:46PM +1000, Devraj Mukherjee wrote:
Hi all,
I am in a bit of a fix here, I have several Guests paravirtualized running on a Dom0 that is currently running the 2.6.18-92.1.22.el5xen Kernel. All my guests (administered by different people) have subsequently been updated to later versions of the kernel (namely 2.6.18-164.el5xen or 2.6.18-128.4.1.el5xen).
Ones that are still running are using the Kernel 2.6.18-128.4.1.el5xen while Dom0 is still booted up into 2.6.18-92.1.22.el5xen
I restarted a Guest that's now trying to boot with the 2.6.18-164.el5xen Kernel and it spits out messages like XENBUS: Waiting for devices to initialise: 295s...290s...
and eventually reboots.
If I try and interrupt the boot and choose a previous version of the Kernel on the Guest, it says the following and fails to boot. The guests use a single disk image that has the SWAP and Root partitions.
Is this happening because the version of the kernel on the Dom0 is way too old for guests to boot? I am just a bit hesitant in rebooting my server for the new kernel to take effect unless I am sure :)
No, that shouldn't be the reason.
The layer between dom0 and guests is Xen hypervisor, not the kernel. Although Redhat ships the hypervisor in the same rpm as the kernel :)
The output looks like the initrd image is broken; it doesn't set up the root device.. does it load the xenblk driver? Does it detect the xvd* disks?
Please paste the full domU kernel boot log..
-- Pasi
Hi Pasi,
On Sun, Sep 27, 2009 at 8:21 PM, Pasi Kärkkäinen pasik@iki.fi wrote:
On Sun, Sep 27, 2009 at 12:46:46PM +1000, Devraj Mukherjee wrote:
Hi all,
I am in a bit of a fix here, I have several Guests paravirtualized running on a Dom0 that is currently running the 2.6.18-92.1.22.el5xen Kernel. All my guests (administered by different people) have subsequently been updated to later versions of the kernel (namely 2.6.18-164.el5xen or 2.6.18-128.4.1.el5xen).
Ones that are still running are using the Kernel 2.6.18-128.4.1.el5xen while Dom0 is still booted up into 2.6.18-92.1.22.el5xen
I restarted a Guest that's now trying to boot with the 2.6.18-164.el5xen Kernel and it spits out messages like XENBUS: Waiting for devices to initialise: 295s...290s...
and eventually reboots.
If I try and interrupt the boot and choose a previous version of the Kernel on the Guest, it says the following and fails to boot. The guests use a single disk image that has the SWAP and Root partitions.
Is this happening because the version of the kernel on the Dom0 is way too old for guests to boot? I am just a bit hesitant in rebooting my server for the new kernel to take effect unless I am sure :)
No, that shouldn't be the reason.
The layer between dom0 and guests is Xen hypervisor, not the kernel. Although Redhat ships the hypervisor in the same rpm as the kernel :)
That's what I thought as well. I did however find the following post while searching for answers http://lists.centos.org/pipermail/centos/2008-January/049757.html
That outlines something similar to what I am experiencing and indicates a reboot of the Dom0 fixed it. I am guessing this is because of Kernel + Driver loading issues.
The output looks like the initrd image is broken; it doesn't set up the root device.. does it load the xenblk driver? Does it detect the xvd* disks?
Is xenblk supposed to be loaded on Dom0?
Please paste the full domU kernel boot log..
The following is my Dom0 xend.log
[2009-09-28 07:48:38 xend 19091] INFO (SrvDaemon:283) Xend Daemon started [2009-09-28 07:48:38 xend 19091] INFO (SrvDaemon:287) Xend changeset: unavailable. [2009-09-28 07:48:38 xend 3549] INFO (SrvDaemon:190) Xend stopped due to signal 15. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 144612597591622L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 1047748L, 'domid': 0, 'max_vcpu_id': 7, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'online_vcpus': 8, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked': 0}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. [2009-09-28 07:48:38 xend 19091] DEBUG (XendDomain:163) number of vcpus to use is 0 [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 649872953225837L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 716584L, 'domid': 11, 'max_vcpu_id': 3, 'crashed': 0, 'running': 0, 'maxmem_kb': 716800L, 'shutdown': 0, 'online_vcpus': 4, 'handle': [158, 197, 14, 207, 118, 131, 228, 14, 11, 141, 80, 155, 130, 15, 78, 19], 'blocked': 1}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 11, UUID 9ec50ecf-7683-e40e-0b8d-509b820f4e13. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 31557343513225L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 614220L, 'domid': 15, 'max_vcpu_id': 0, 'crashed': 0, 'running': 0, 'maxmem_kb': 614400L, 'shutdown': 0, 'online_vcpus': 1, 'handle': [205, 119, 123, 10, 116, 195, 147, 157, 102, 33, 48, 200, 229, 25, 31, 67], 'blocked': 1}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 15, UUID cd777b0a-74c3-939d-6621-30c8e5191f43. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 439665875780158L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 1433392L, 'domid': 19, 'max_vcpu_id': 3, 'crashed': 0, 'running': 0, 'maxmem_kb': 1433600L, 'shutdown': 0, 'online_vcpus': 4, 'handle': [5, 72, 139, 170, 228, 55, 87, 82, 223, 36, 98, 155, 30, 97, 160, 1], 'blocked': 1}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 19, UUID 05488baa-e437-5752-df24-629b1e61a001. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 46317871738603L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 1048340L, 'domid': 20, 'max_vcpu_id': 3, 'crashed': 0, 'running': 0, 'maxmem_kb': 1048576L, 'shutdown': 0, 'online_vcpus': 4, 'handle': [30, 213, 122, 77, 195, 98, 64, 180, 114, 126, 54, 154, 27, 216, 152, 159], 'blocked': 1}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 20, UUID 1ed57a4d-c362-40b4-727e-369a1bd8989f. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 39789190060430L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 818976L, 'domid': 23, 'max_vcpu_id': 3, 'crashed': 0, 'running': 0, 'maxmem_kb': 819200L, 'shutdown': 0, 'online_vcpus': 4, 'handle': [32, 236, 171, 210, 127, 13, 153, 208, 190, 42, 45, 92, 227, 168, 64, 150], 'blocked': 1}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 23, UUID 20ecabd2-7f0d-99d0-be2a-2d5ce3a84096. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 50167261004541L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 1048392L, 'domid': 24, 'max_vcpu_id': 3, 'crashed': 0, 'running': 0, 'maxmem_kb': 1048576L, 'shutdown': 0, 'online_vcpus': 4, 'handle': [158, 249, 96, 169, 101, 191, 89, 222, 50, 232, 66, 187, 99, 247, 51, 94], 'blocked': 1}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 24, UUID 9ef960a9-65bf-59de-32e8-42bb63f7335e. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:222) XendDomainInfo.recreate({'paused': 0, 'cpu_time': 17188583788408L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'mem_kb': 946040L, 'domid': 25, 'max_vcpu_id': 3, 'crashed': 0, 'running': 0, 'maxmem_kb': 946176L, 'shutdown': 0, 'online_vcpus': 4, 'handle': [43, 153, 55, 197, 66, 189, 21, 17, 17, 147, 2, 101, 15, 21, 17, 36], 'blocked': 1}) [2009-09-28 07:48:38 xend.XendDomainInfo 19091] INFO (XendDomainInfo:234) Recreating domain 25, UUID 2b9937c5-42bd-1511-1193-02650f151124. [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:48:38 xend 19091] INFO (SrvServer:116) unix path=/var/lib/xend/xend-socket [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name', 'svn'], ['memory', '512'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 4], ['bootloader', '/usr/bin/pygrub'], ['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.3XC1Fl'], ['kernel', '/var/lib/xen/boot_kernel.RfjOtc'], ['args', 'ro root=/dev/VolGroup00/LogVol00 console=xvc0']]], ['device', ['tap', ['uname', 'tap:aio:/srv/xen/svn.img'], ['dev', 'xvda'], ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr0']]]]) [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:312) parseConfig: config is ['vm', ['name', 'svn'], ['memory', '512'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 4], ['bootloader', '/usr/bin/pygrub'], ['image', ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.3XC1Fl'], ['kernel', '/var/lib/xen/boot_kernel.RfjOtc'], ['args', 'ro root=/dev/VolGroup00/LogVol00 console=xvc0']]], ['device', ['tap', ['uname', 'tap:aio:/srv/xen/svn.img'], ['dev', 'xvda'], ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr0']]]] [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:417) parseConfig: result is {'shadow_memory': None, 'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot': 'restart', 'localtime': None, 'image': ['linux', ['ramdisk', '/var/lib/xen/boot_ramdisk.3XC1Fl'], ['kernel', '/var/lib/xen/boot_kernel.RfjOtc'], ['args', 'ro root=/dev/VolGroup00/LogVol00 console=xvc0']], 'on_poweroff': None, 'bootloader_args': None, 'cpus': None, 'name': 'svn', 'backend': [], 'vcpus': 4, 'cpu_weight': None, 'features': None, 'vcpu_avail': None, 'memory': 512, 'device': [('tap', ['tap', ['uname', 'tap:aio:/srv/xen/svn.img'], ['dev', 'xvda'], ['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr0']])], 'bootloader': '/usr/bin/pygrub', 'cpu': None, 'maxmem': None} [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1427) XendDomainInfo.construct: None [2009-09-28 07:49:11 xend 19091] DEBUG (balloon:143) Balloon: 574488 KiB free; need 2048; done. [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1475) XendDomainInfo.initDomain: 63 1.0 [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1514) _initDomain:shadow_memory=0x0, maxmem=0x200, memory=0x200. [2009-09-28 07:49:11 xend 19091] DEBUG (balloon:143) Balloon: 574480 KiB free; need 524288; done. [2009-09-28 07:49:11 xend 19091] INFO (image:137) buildDomain os=linux dom=63 vcpus=4 [2009-09-28 07:49:11 xend 19091] DEBUG (image:206) domid = 63 [2009-09-28 07:49:11 xend 19091] DEBUG (image:207) memsize = 512 [2009-09-28 07:49:11 xend 19091] DEBUG (image:208) image = /var/lib/xen/boot_kernel.RfjOtc [2009-09-28 07:49:11 xend 19091] DEBUG (image:209) store_evtchn = 1 [2009-09-28 07:49:11 xend 19091] DEBUG (image:210) console_evtchn = 2 [2009-09-28 07:49:11 xend 19091] DEBUG (image:211) cmdline = ro root=/dev/VolGroup00/LogVol00 console=xvc0 [2009-09-28 07:49:11 xend 19091] DEBUG (image:212) ramdisk = /var/lib/xen/boot_ramdisk.3XC1Fl [2009-09-28 07:49:11 xend 19091] DEBUG (image:213) vcpus = 4 [2009-09-28 07:49:11 xend 19091] DEBUG (image:214) features = [2009-09-28 07:49:11 xend 19091] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda' [2009-09-28 07:49:11 xend 19091] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda' [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:110) DevController: writing {'virtual-device': '51712', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/tap/63/51712'} to /local/domain/63/device/vbd/51712. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:112) DevController: writing {'domain': 'svn', 'frontend': '/local/domain/63/device/vbd/51712', 'format': 'raw', 'dev': 'xvda', 'state': '1', 'params': 'aio:/srv/xen/svn.img', 'mode': 'w', 'online': '1', 'frontend-id': '63', 'type': 'tap'} to /local/domain/0/backend/tap/63/51712. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:110) DevController: writing {'mac': '00:16:3e:0d:86:28', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/63/0'} to /local/domain/63/device/vif/0. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:112) DevController: writing {'bridge': 'xenbr0', 'domain': 'svn', 'handle': '0', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/63/device/vif/0', 'mac': '00:16:3e:0d:86:28', 'online': '1', 'frontend-id': '63'} to /local/domain/0/backend/vif/63/0. [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:797) Storing VM details: {'shadow_memory': '0', 'uuid': '6ce0a479-9e75-5767-5f7b-4bf325aac895', 'on_reboot': 'restart', 'start_time': '1254088151.53', 'on_poweroff': 'destroy', 'name': 'svn', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'memory': '512', 'on_crash': 'restart', 'image': "(linux (ramdisk /var/lib/xen/boot_ramdisk.3XC1Fl) (kernel /var/lib/xen/boot_kernel.RfjOtc) (args 'ro root=/dev/VolGroup00/LogVol00 console=xvc0'))", 'maxmem': '512'} [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:832) Storing domain details: {'console/ring-ref': '425496', 'console/port': '2', 'cpu/3/availability': 'online', 'name': 'svn', 'console/limit': '1048576', 'cpu/2/availability': 'online', 'vm': '/vm/6ce0a479-9e75-5767-5f7b-4bf325aac895', 'domid': '63', 'cpu/0/availability': 'online', 'memory/target': '524288', 'store/ring-ref': '372349', 'cpu/1/availability': 'online', 'store/port': '1'} [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices vif. [2009-09-28 07:49:11 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1036) XendDomainInfo.handleShutdownWatch [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:160) Waiting for 0. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:496) hotplugStatusCallback /local/domain/0/backend/vif/63/0/hotplug-status. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:496) hotplugStatusCallback /local/domain/0/backend/vif/63/0/hotplug-status. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:510) hotplugStatusCallback 1. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices usb. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices vbd. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices irq. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices vkbd. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices vfb. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices pci. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices ioports. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices tap. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:160) Waiting for 51712. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:496) hotplugStatusCallback /local/domain/0/backend/tap/63/51712/hotplug-status. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:510) hotplugStatusCallback 1. [2009-09-28 07:49:11 xend 19091] DEBUG (DevController:154) Waiting for devices vtpm. [2009-09-28 07:49:11 xend 19091] INFO (XendDomain:380) Domain svn (63) unpaused. [2009-09-28 07:49:32 xend.XendDomainInfo 19091] DEBUG (XendDomainInfo:1637) XendDomainInfo.destroy: domid=63
and this is my boot log for the DomU
Using config file "/etc/xen/svn". Started domain svn Bootdata ok (command line is ro root=/dev/VolGroup00/LogVol00 console=xvc0) Linux version 2.6.18-164.el5xen (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 04:03:03 EDT 2009 BIOS-provided physical RAM map: Xen: 0000000000000000 - 0000000020800000 (usable) No mptable found. Built 1 zonelists. Total pages: 133120 Kernel command line: ro root=/dev/VolGroup00/LogVol00 console=xvc0 Initializing CPU#0 PID hash table entries: 4096 (order: 12, 32768 bytes) Xen reported: 2493.748 MHz processor. Console: colour dummy device 80x25 Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) Software IO TLB disabled Memory: 499456k/532480k available (2490k kernel code, 24144k reserved, 1388k data, 184k init) Calibrating delay using timer specific routine.. 6238.93 BogoMIPS (lpj=12477871) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 6144K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 2 (SMP-)alternatives turned off Brought up 1 CPUs checking if image is initramfs... it is Grant table initialized NET: Registered protocol family 16 Initializing CPU#1 migration_cost=1186 Initializing CPU#2 migration_cost=1186 migration_cost=Initializing CPU#3 1186 Brought up 4 CPUs PCI: setting up Xen PCI frontend stub ACPI: Interpreter disabled. Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI: disabled xen_mem: Initialising balloon driver. usbcore: registered new driver usbfs usbcore: registered new driver hub PCI: System does not support PCI PCI: System does not support PCI NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default NET: Registered protocol family 2 IP route cache hash table entries: 32768 (order: 6, 262144 bytes) TCP established hash table entries: 131072 (order: 9, 2097152 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) TCP: Hash tables configured (established 131072 bind 65536) TCP reno registered audit: initializing netlink socket (disabled) type=2000 audit(1254090517.635:1): initialized VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Initializing Cryptographic API alg: No test for crc32c (crc32c-generic) ksign: Installing public key data Loading keyring - Added public key 57F09578A977B70E - User ID: CentOS (Kernel Module GPG key) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 rtc: IRQ 8 is not free. Non-volatile memory driver v1.2 Linux agpgart interface v0.101 (c) Dave Jones brd: module loaded Xen virtual console successfully installed as xvc0 Bootdata ok (command line is ro root=/dev/VolGroup00/LogVol00 console=xvc0) Linux version 2.6.18-164.el5xen (mockbuild@builder10.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Thu Sep 3 04:03:03 EDT 2009 BIOS-provided physical RAM map: Xen: 0000000000000000 - 0000000020800000 (usable) No mptable found. Built 1 zonelists. Total pages: 133120 Kernel command line: ro root=/dev/VolGroup00/LogVol00 console=xvc0 Initializing CPU#0 PID hash table entries: 4096 (order: 12, 32768 bytes) Xen reported: 2493.748 MHz processor. Console: colour dummy device 80x25 Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) Software IO TLB disabled Memory: 499456k/532480k available (2490k kernel code, 24144k reserved, 1388k data, 184k init) Calibrating delay using timer specific routine.. 6238.93 BogoMIPS (lpj=12477871) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 256 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 6144K CPU: Physical Processor ID: 1 CPU: Processor Core ID: 2 (SMP-)alternatives turned off Brought up 1 CPUs checking if image is initramfs... it is Grant table initialized NET: Registered protocol family 16 Initializing CPU#1 migration_cost=1186 Initializing CPU#2 migration_cost=1186 migration_cost=Initializing CPU#3 1186 Brought up 4 CPUs PCI: setting up Xen PCI frontend stub ACPI: Interpreter disabled. Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI: disabled xen_mem: Initialising balloon driver. usbcore: registered new driver usbfs usbcore: registered new driver hub PCI: System does not support PCI PCI: System does not support PCI NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default NET: Registered protocol family 2 IP route cache hash table entries: 32768 (order: 6, 262144 bytes) TCP established hash table entries: 131072 (order: 9, 2097152 bytes) TCP bind hash table entries: 65536 (order: 8, 1048576 bytes) TCP: Hash tables configured (established 131072 bind 65536) TCP reno registered audit: initializing netlink socket (disabled) type=2000 audit(1254090517.635:1): initialized VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 512 (order 0, 4096 bytes) Initializing Cryptographic API alg: No test for crc32c (crc32c-generic) ksign: Installing public key data Loading keyring - Added public key 57F09578A977B70E - User ID: CentOS (Kernel Module GPG key) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 rtc: IRQ 8 is not free. Non-volatile memory driver v1.2 Linux agpgart interface v0.101 (c) Dave Jones brd: module loaded Xen virtual console successfully installed as xvc0 Event-channel device installed. Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx ide-floppy driver 0.99.newide usbcore: registered new driver hiddev usbcore: registered new driver usbhid drivers/usb/input/hid-core.c: v2.6:USB HID core driver PNP: No PS/2 controller found. Probing ports directly. i8042.c: No controller found. mice: PS/2 mouse device common for all mice md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 md: bitmap version 4.39 TCP bic registered Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 17 XENBUS: Device with no driver: device/vbd/51712 XENBUS: Device with no driver: device/vif/0 Initalizing network drop monitor service Write protecting the kernel read-only data: 478k Red Hat nash version 5.1.19.6 starting Mounting proc filesystem Mounting sysfs filesystem Creating /dev Creating initial device nodes Setting up hotplug. Creating block device nodes. Loading ehci-hcd.ko module Loading ohci-hcd.ko module Loading uhci-hcd.ko module USB Universal Host Controller Interface driver v3.0 Loading jbd.ko module Loading ext3.ko module Loading xenblk.ko module
On Mon, Sep 28, 2009 at 08:32:12AM +1000, Devraj Mukherjee wrote:
Is this happening because the version of the kernel on the Dom0 is way too old for guests to boot? I am just a bit hesitant in rebooting my server for the new kernel to take effect unless I am sure :)
No, that shouldn't be the reason.
The layer between dom0 and guests is Xen hypervisor, not the kernel. Although Redhat ships the hypervisor in the same rpm as the kernel :)
That's what I thought as well. I did however find the following post while searching for answers http://lists.centos.org/pipermail/centos/2008-January/049757.html
That outlines something similar to what I am experiencing and indicates a reboot of the Dom0 fixed it. I am guessing this is because of Kernel + Driver loading issues.
The output looks like the initrd image is broken; it doesn't set up the root device.. does it load the xenblk driver? Does it detect the xvd* disks?
Is xenblk supposed to be loaded on Dom0?
No, in the guest kernel.
and this is my boot log for the DomU
initrd execution starts here..
Write protecting the kernel read-only data: 478k Red Hat nash version 5.1.19.6 starting Mounting proc filesystem Mounting sysfs filesystem Creating /dev Creating initial device nodes Setting up hotplug. Creating block device nodes. Loading ehci-hcd.ko module Loading ohci-hcd.ko module Loading uhci-hcd.ko module USB Universal Host Controller Interface driver v3.0 Loading jbd.ko module Loading ext3.ko module Loading xenblk.ko module
Ok, so it hangs at loading xenblk? No further output?
-- Pasi
Hi,
where does you root-device reside on? Do you use sda or hda?
Can you post your DomU-config (without comments) here?
What messages do you get before LVM complains? It might be that you don't have the rigth drivers in your DomU-initrd (any more)...
Kind regards,
Nils
-----Original Message----- From: centos-virt-bounces@centos.org [mailto:centos-virt-bounces@centos.org] On Behalf Of Devraj Mukherjee Sent: Sunday, September 27, 2009 4:47 AM To: Discussion about the virtualization on CentOS Subject: [CentOS-virt] Domain0 with Kernel 2.6.18-92.1.22.el5xen fails toboot Guests with Kernel 2.6.18-164.el5xen
Hi all,
I am in a bit of a fix here, I have several Guests paravirtualized running on a Dom0 that is currently running the 2.6.18-92.1.22.el5xen Kernel. All my guests (administered by different people) have subsequently been updated to later versions of the kernel (namely 2.6.18-164.el5xen or 2.6.18-128.4.1.el5xen).
Ones that are still running are using the Kernel 2.6.18-128.4.1.el5xen while Dom0 is still booted up into 2.6.18-92.1.22.el5xen
I restarted a Guest that's now trying to boot with the 2.6.18-164.el5xen Kernel and it spits out messages like XENBUS: Waiting for devices to initialise: 295s...290s...
and eventually reboots.
If I try and interrupt the boot and choose a previous version of the Kernel on the Guest, it says the following and fails to boot. The guests use a single disk image that has the SWAP and Root partitions.
Is this happening because the version of the kernel on the Dom0 is way too old for guests to boot? I am just a bit hesitant in rebooting my server for the new kernel to take effect unless I am sure :)
device-mapper: dm-raid45: initialized v0.2429 Scanning and configuring dmraid supported devices Scanning logical volumes Reading all physical volumes. This may take a while... Activating logical volumes Volume group "VolGroup00" not found Creating root device. Mounting root filesystem. mount: could not find filesystem '/dev/root' Setting up other filesystems. Setting up new root fs setuproot: moving /dev failed: No such file or directory no fstab.sys, mounting internal defaults setuproot: error mounting /proc: No such file or directory setuproot: error mounting /sys: No such file or directory Switching to new root and running init. unmounting old /dev unmounting old /proc unmounting old /sys switchroot: mount failed: No such file or directory Kernel panic - not syncing: Attempted to kill init!
-- "The secret impresses no-one, the trick you use it for is everything"
- Alfred Borden (The Prestiege)
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt