First let me say that I'm not a sysadmin, but am simply wearing that hat this week so please excuse my ignorance. I need to temporarily move some virtual servers from a CentOS-KVM platform to a CentOS-XEN platform while I do some upgrades to the CentOS box. I've created a local LV, and used DD and SCP to transfer the block device from the VKM machine to the XEN machine. For quite a while I struggled with the "Error: (2, 'Invalid kernel', 'xc_dom_parse_elf_kernel: ELF image has no shstrtab\n')" error but thanks to Nick Couchman from http://lists.xensource.com/archives/html/xen-users/2008-03/msg00603.html, I've passed that. Now I'm stuck with a kernel panic situation. Unfortunately the kernel panic error doesn't appear using "xm console" and flashes so quickly through "virt-viewer" that I can't get any details. I think it may have to do with the differing file structures on the 2 Host machines. The KVM machine mounts / on /dev/mapper/vg00-root whereas the XEN machine mounts / on /dev/mapper/VMM01-KVMvg00-c5root00:
[root@VMM01-KVM ~]# mount /dev/mapper/vg00-root on / type ext3 (rw)
[root@VMM03-XEN xen]# mount /dev/mapper/VMM01-KVMvg00-c5root00 on / type ext3 (rw)
Interesting log file entries: [2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda' [2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
-------------------------------------------------------------------------------- The domU is configured thusly: [root@VMM03-XEN xen]# cat /etc/xen/guest02 # Automatically generated xen config file name = "guest02" memory = "1024" disk = [ 'phy:/dev/VMM01-KVMvg00/guest02,xvda,w', ] vif = [ 'mac=00:16:3e:5c:f1:dd, bridge=xenbr0', ] vfb = ["type=vnc,vncunused=1"] #bootloader="/usr/bin/pygrub" kernel = "/boot/vmlinuz-2.6.18-164.6.1.el5xen" ramdisk = "/boot/initrd-2.6.18-164.6.1.el5xen.img" vcpus=1 on_reboot = 'restart' on_crash = 'restart' #root = "/dev/VMM01-KVMvg00/c5root00 ro" #extra = "single selinux=0" extra = "ro"
--------------------------------------------------------------------------------
From the domU instance running on the KVM box:
[root@guest02 ~]# cat /etc/fstab /dev/vg00/root / ext3 defaults 1 1 LABEL=/boot /boot ext3 defaults 1 2 tmpfs /dev/shm tmpfs defaults 0 0 devpts /dev/pts devpts gid=5,mode=620 0 0 sysfs /sys sysfs defaults 0 0 proc /proc proc defaults 0 0 /dev/vg00/swap swap swap defaults 0 0
[root@guest02 ~]# cat /etc/gr group group- grub.conf
-------------------------------------------------------------------------------- [root@guest02 ~]# cat /etc/grub.conf # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro root=/dev/vg00/root divider=10 notsc # initrd /initrd-version.img #boot=/dev/vda default=0 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.18-164.el5) root (hd0,0) kernel /vmlinuz-2.6.18-164.el5 ro root=/dev/vg00/root divider=10 notsc initrd /initrd-2.6.18-164.el5.img title CentOS (2.6.18-128.7.1.el5) root (hd0,0) kernel /vmlinuz-2.6.18-128.7.1.el5 ro root=/dev/vg00/root divider=10 notsc initrd /initrd-2.6.18-128.7.1.el5.img title CentOS (2.6.18-128.4.1.el5) root (hd0,0) kernel /vmlinuz-2.6.18-128.4.1.el5 ro root=/dev/vg00/root divider=10 notsc initrd /initrd-2.6.18-128.4.1.el5.img title CentOS (2.6.18-128.2.1.el5) root (hd0,0) kernel /vmlinuz-2.6.18-128.2.1.el5 ro root=/dev/vg00/root divider=10 notsc initrd /initrd-2.6.18-128.2.1.el5.img title CentOS (2.6.18-128.1.10.el5) root (hd0,0) kernel /vmlinuz-2.6.18-128.1.10.el5 ro root=/dev/vg00/root divider=10 notsc initrd /initrd-2.6.18-128.1.10.el5.img
-------------------------------------------------------------------------------- Dom-0 Log files: [root@VMM03-XEN xen]# tail -f /var/log/xen/xend.log [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:206) XendDomainInfo.create(['vm', ['name', 'guest02'], ['memory', '1024'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.18-164.6.1.el5xen'], ['ramdisk', '/boot/initrd-2.6.18-164.6.1.el5xen.img'], ['args', 'ro']]], ['device', ['vbd', ['uname', 'phy:/dev/VMM01-KVMvg00/guest02'], ['dev', 'xvda'], ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:5c:f1:dd']]], ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type', 'vnc'], ['xauthority', '/root/.Xauthority']]]]) [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:328) parseConfig: config is ['vm', ['name', 'guest02'], ['memory', '1024'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.18-164.6.1.el5xen'], ['ramdisk', '/boot/initrd-2.6.18-164.6.1.el5xen.img'], ['args', 'ro']]], ['device', ['vbd', ['uname', 'phy:/dev/VMM01-KVMvg00/guest02'], ['dev', 'xvda'], ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:5c:f1:dd']]], ['device', ['vkbd']], ['device', ['vfb', ['vncunused', '1'], ['type', 'vnc'], ['xauthority', '/root/.Xauthority']]]] [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:445) parseConfig: result is {'features': None, 'image': ['linux', ['kernel', '/boot/vmlinuz-2.6.18-164.6.1.el5xen'], ['ramdisk', '/boot/initrd-2.6.18-164.6.1.el5xen.img'], ['args', 'ro']], 'cpus': None, 'vcpu_avail': None, 'backend': [], 'uuid': None, 'on_reboot': 'restart', 'cpu_weight': None, 'memory': 1024, 'cpu_cap': None, 'localtime': None, 'timer_mode': None, 'start_time': None, 'on_poweroff': None, 'on_crash': 'restart', 'device': [('vbd', ['vbd', ['uname', 'phy:/dev/VMM01-KVMvg00/guest02'], ['dev', 'xvda'], ['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:5c:f1:dd']]), ('vkbd', ['vkbd']), ('vfb', ['vfb', ['vncunused', '1'], ['type', 'vnc'], ['xauthority', '/root/.Xauthority']])], 'bootloader': None, 'maxmem': None, 'shadow_memory': None, 'name': 'guest02', 'bootloader_args': None, 'vcpus': 1, 'cpu': None} [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1774) XendDomainInfo.construct: None [2009-11-06 10:27:26 xend 5112] DEBUG (balloon:145) Balloon: 3145436 KiB free; need 4096; done. [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1900) XendDomainInfo.initDomain: 50 256 [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1941) _initDomain:shadow_memory=0x0, maxmem=0x400, memory=0x400. [2009-11-06 10:27:26 xend 5112] DEBUG (balloon:145) Balloon: 3145436 KiB free; need 1048576; done. [2009-11-06 10:27:26 xend 5112] INFO (image:139) buildDomain os=linux dom=50 vcpus=1 [2009-11-06 10:27:26 xend 5112] DEBUG (image:208) domid = 50 [2009-11-06 10:27:26 xend 5112] DEBUG (image:209) memsize = 1024 [2009-11-06 10:27:26 xend 5112] DEBUG (image:210) image = /boot/vmlinuz-2.6.18-164.6.1.el5xen [2009-11-06 10:27:26 xend 5112] DEBUG (image:211) store_evtchn = 1 [2009-11-06 10:27:26 xend 5112] DEBUG (image:212) console_evtchn = 2 [2009-11-06 10:27:26 xend 5112] DEBUG (image:213) cmdline = ro [2009-11-06 10:27:26 xend 5112] DEBUG (image:214) ramdisk = /boot/initrd-2.6.18-164.6.1.el5xen.img [2009-11-06 10:27:26 xend 5112] DEBUG (image:215) vcpus = 1 [2009-11-06 10:27:26 xend 5112] DEBUG (image:216) features = [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController: writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vkbd/50/0'} to /local/domain/50/device/vkbd/0. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController: writing {'frontend-id': '50', 'domain': 'guest02', 'frontend': '/local/domain/50/device/vkbd/0', 'state': '1', 'online': '1'} to /local/domain/0/backend/vkbd/50/0. [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:633) Checking for duplicate for uname: /dev/VMM01-KVMvg00/guest02 [phy:/dev/VMM01-KVMvg00/guest02], dev: xvda, mode: w [2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda' [2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device number for xvda: [Errno 2] No such file or directory: '/dev/xvda' [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController: writing {'virtual-device': '51712', 'device-type': 'disk', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/50/51712'} to /local/domain/50/device/vbd/51712. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController: writing {'domain': 'guest02', 'frontend': '/local/domain/50/device/vbd/51712', 'format': 'raw', 'dev': 'xvda', 'state': '1', 'params': '/dev/VMM01-KVMvg00/guest02', 'mode': 'w', 'online': '1', 'frontend-id': '50', 'type': 'phy'} to /local/domain/0/backend/vbd/50/51712. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController: writing {'mac': '00:16:3e:5c:f1:dd', 'handle': '0', 'protocol': 'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/50/0'} to /local/domain/50/device/vif/0. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController: writing {'bridge': 'xenbr0', 'domain': 'guest02', 'handle': '0', 'script': '/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend': '/local/domain/50/device/vif/0', 'mac': '00:16:3e:5c:f1:dd', 'online': '1', 'frontend-id': '50'} to /local/domain/0/backend/vif/50/0. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController: writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/50/0'} to /local/domain/50/device/vfb/0. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController: writing {'vncunused': '1', 'domain': 'guest02', 'frontend': '/local/domain/50/device/vfb/0', 'xauthority': '/root/.Xauthority', 'state': '1', 'online': '1', 'frontend-id': '50', 'type': 'vnc'} to /local/domain/0/backend/vfb/50/0. [2009-11-06 10:27:26 xend 5112] DEBUG (vfbif:11) Spawn: ['/usr/lib/xen/bin/qemu-dm', '-M', 'xenpv', '-d', '50', '-domain-name', 'guest02', '-vnc', '127.0.0.1:0', '-vncunused', '-k', 'en-us'] [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:992) Storing VM details: {'shadow_memory': '0', 'uuid': '21fa15ea-febf-1cbd-6360-cfa02eb73d8c', 'on_reboot': 'restart', 'start_time': '1257532046.49', 'on_poweroff': 'destroy', 'name': 'guest02', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory': '1024', 'on_crash': 'restart', 'image': '(linux (kernel /boot/vmlinuz-2.6.18-164.6.1.el5xen) (ramdisk /boot/initrd-2.6.18-164.6.1.el5xen.img) (args ro))', 'maxmem': '1024'} [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1027) Storing domain details: {'console/ring-ref': '1084703', 'console/port': '2', 'name': 'guest02', 'console/limit': '1048576', 'vm': '/vm/21fa15ea-febf-1cbd-6360-cfa02eb73d8c', 'domid': '50', 'cpu/0/availability': 'online', 'memory/target': '1048576', 'store/ring-ref': '1055232', 'store/port': '1'} [2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1249) XendDomainInfo.handleShutdownWatch [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices vif. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 0. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vif/50/0/hotplug-status. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vif/50/0/hotplug-status. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices usb. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices vbd. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 51712. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vbd/50/51712/hotplug-status. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices irq. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices vkbd. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 0. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vkbd/50/0/hotplug-status. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices vfb. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 0. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509) hotplugStatusCallback /local/domain/0/backend/vfb/50/0/hotplug-status. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523) hotplugStatusCallback 1. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices pci. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices ioports. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices tap. [2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for devices vtpm. [2009-11-06 10:27:26 xend 5112] INFO (XendDomain:387) Domain guest02 (50) unpaused.
[root@VMM03-XEN xen]# xm create guest02 ; xm console guest02 Using config file "./guest02". Started domain guest02 Linux version 2.6.18-164.6.1.el5xen (mockbuild@builder16.centos.org) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Nov 3 17:53:47 EST 2009 BIOS-provided physical RAM map: Xen: 0000000000000000 - 0000000040800000 (usable) 304MB HIGHMEM available. 727MB LOWMEM available. NX (Execute Disable) protection: active ACPI in unprivileged domain disabled Built 1 zonelists. Total pages: 264192 Kernel command line: ro Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 CPU 0 irqstacks, hard=c0750000 soft=c0730000 PID hash table entries: 4096 (order: 12, 16384 bytes) Xen reported: 2992.688 MHz processor. Console: colour dummy device 80x25 Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) Software IO TLB disabled vmalloc area: ee000000-f4ffe000, maxmem 2d7fe000 Memory: 1025064k/1056768k available (2162k kernel code, 22592k reserved, 887k data, 176k init, 311304k highmem) Checking if this processor honours the WP bit even in supervisor mode... Ok. Calibrating delay using timer specific routine.. 7493.78 BogoMIPS (lpj=14987562) Security Framework v1.0.0 initialized SELinux: Initializing. selinux_register_security: Registering secondary module capability Capability LSM initialized as secondary Mount-cache hash table entries: 512 CPU: Trace cache: 12K uops, L1 D cache: 16K CPU: L2 cache: 2048K Checking 'hlt' instruction... OK. SMP alternatives: switching to UP code Freeing SMP alternatives: 13k freed Brought up 1 CPUs checking if image is initramfs... it is Freeing initrd memory: 6980k freed Grant table initialized NET: Registered protocol family 16 Brought up 1 CPUs PCI: setting up Xen PCI frontend stub ACPI: Interpreter disabled. Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI: disabled xen_mem: Initialising balloon driver. usbcore: registered new driver usbfs usbcore: registered new driver hub PCI: System does not support PCI PCI: System does not support PCI NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default NET: Registered protocol family 2 IP route cache hash table entries: 32768 (order: 5, 131072 bytes) TCP established hash table entries: 131072 (order: 8, 1048576 bytes) TCP bind hash table entries: 65536 (order: 7, 524288 bytes) TCP: Hash tables configured (established 131072 bind 65536) TCP reno registered audit: initializing netlink socket (disabled) type=2000 audit(1257533537.470:1): initialized highmem bounce pool size: 64 pages VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Initializing Cryptographic API alg: No test for crc32c (crc32c-generic) ksign: Installing public key data Loading keyring - Added public key 5CA5E7AEC7491D45 - User ID: CentOS (Kernel Module GPG key) io scheduler noop registered io scheduler anticipatory registered io scheduler deadline registered io scheduler cfq registered (default) pci_hotplug: PCI Hot Plug PCI Core version: 0.5 rtc: IRQ 8 is not free. Non-volatile memory driver v1.2 Linux agpgart interface v0.101 (c) Dave Jones brd: module loaded Xen virtual console successfully installed as xvc0 Event-channel device installed. Console: switching to colour frame buffer device 100x37 input: Xen Virtual Keyboard/Mouse as /class/input/input0 Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2 ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx ide-floppy driver 0.99.newide usbcore: registered new driver hiddev usbcore: registered new driver usbhid drivers/usb/input/hid-core.c: v2.6:USB HID core driver PNP: No PS/2 controller found. Probing ports directly. i8042.c: No controller found. mice: PS/2 mouse device common for all mice md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27 md: bitmap version 4.39 TCP bic registered Initializing IPsec netlink socket NET: Registered protocol family 1 NET: Registered protocol family 17 Using IPI No-Shortcut mode XENBUS: Device with no driver: device/vbd/51712 XENBUS: Device with no driver: device/vif/0 Initalizing network drop monitor service Freeing unused kernel memory: 176k freed Write protecting the kernel read-only data: 390k USB Universal Host Controller Interface driver v3.0 megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006) SCSI subsystem initialized megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006) device-mapper: uevent: version 1.0.3 device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised: dm-devel@redhat.com device-mapper: dm-raid45: initialized v0.2594l ##### using virt-viewer to view the console I see one more line: #### waiting for driver initialization...
Grovellingly yours, with bruised forehead -Chris