[CentOS-virt] Migrating from KVM to XEN - kernel panic
Christopher Hunt
dharmachris at gmail.com
Fri Nov 6 19:49:41 UTC 2009
First let me say that I'm not a sysadmin, but am simply wearing that hat
this week so please excuse my ignorance. I need to temporarily move some
virtual servers from a CentOS-KVM platform to a CentOS-XEN platform while I
do some upgrades to the CentOS box. I've created a local LV, and used DD
and SCP to transfer the block device from the VKM machine to the XEN
machine. For quite a while I struggled with the "Error: (2, 'Invalid
kernel', 'xc_dom_parse_elf_kernel: ELF image has no shstrtab\n')" error but
thanks to Nick Couchman from
http://lists.xensource.com/archives/html/xen-users/2008-03/msg00603.html,
I've passed that. Now I'm stuck with a kernel panic situation.
Unfortunately the kernel panic error doesn't appear using "xm console" and
flashes so quickly through "virt-viewer" that I can't get any details. I
think it may have to do with the differing file structures on the 2 Host
machines. The KVM machine mounts / on /dev/mapper/vg00-root whereas the XEN
machine mounts / on /dev/mapper/VMM01-KVMvg00-c5root00:
[root at VMM01-KVM ~]# mount
/dev/mapper/vg00-root on / type ext3 (rw)
[root at VMM03-XEN xen]# mount
/dev/mapper/VMM01-KVMvg00-c5root00 on / type ext3 (rw)
Interesting log file entries:
[2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device
number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device
number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
--------------------------------------------------------------------------------
The domU is configured thusly:
[root at VMM03-XEN xen]# cat /etc/xen/guest02
# Automatically generated xen config file
name = "guest02"
memory = "1024"
disk = [ 'phy:/dev/VMM01-KVMvg00/guest02,xvda,w', ]
vif = [ 'mac=00:16:3e:5c:f1:dd, bridge=xenbr0', ]
vfb = ["type=vnc,vncunused=1"]
#bootloader="/usr/bin/pygrub"
kernel = "/boot/vmlinuz-2.6.18-164.6.1.el5xen"
ramdisk = "/boot/initrd-2.6.18-164.6.1.el5xen.img"
vcpus=1
on_reboot = 'restart'
on_crash = 'restart'
#root = "/dev/VMM01-KVMvg00/c5root00 ro"
#extra = "single selinux=0"
extra = "ro"
--------------------------------------------------------------------------------
>From the domU instance running on the KVM box:
[root at guest02 ~]# cat /etc/fstab
/dev/vg00/root / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/vg00/swap swap swap defaults 0 0
[root at guest02 ~]# cat /etc/gr
group group- grub.conf
--------------------------------------------------------------------------------
[root at guest02 ~]# cat /etc/grub.conf
# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE: You have a /boot partition. This means that
# all kernel and initrd paths are relative to /boot/, eg.
# root (hd0,0)
# kernel /vmlinuz-version ro root=/dev/vg00/root divider=10 notsc
# initrd /initrd-version.img
#boot=/dev/vda
default=0
timeout=5
splashimage=(hd0,0)/grub/splash.xpm.gz
hiddenmenu
title CentOS (2.6.18-164.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-164.el5 ro root=/dev/vg00/root divider=10
notsc
initrd /initrd-2.6.18-164.el5.img
title CentOS (2.6.18-128.7.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.7.1.el5 ro root=/dev/vg00/root divider=10
notsc
initrd /initrd-2.6.18-128.7.1.el5.img
title CentOS (2.6.18-128.4.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.4.1.el5 ro root=/dev/vg00/root divider=10
notsc
initrd /initrd-2.6.18-128.4.1.el5.img
title CentOS (2.6.18-128.2.1.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.2.1.el5 ro root=/dev/vg00/root divider=10
notsc
initrd /initrd-2.6.18-128.2.1.el5.img
title CentOS (2.6.18-128.1.10.el5)
root (hd0,0)
kernel /vmlinuz-2.6.18-128.1.10.el5 ro root=/dev/vg00/root
divider=10 notsc
initrd /initrd-2.6.18-128.1.10.el5.img
--------------------------------------------------------------------------------
Dom-0 Log files:
[root at VMM03-XEN xen]# tail -f /var/log/xen/xend.log
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:206)
XendDomainInfo.create(['vm', ['name', 'guest02'], ['memory', '1024'],
['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['image',
['linux', ['kernel', '/boot/vmlinuz-2.6.18-164.6.1.el5xen'], ['ramdisk',
'/boot/initrd-2.6.18-164.6.1.el5xen.img'], ['args', 'ro']]], ['device',
['vbd', ['uname', 'phy:/dev/VMM01-KVMvg00/guest02'], ['dev', 'xvda'],
['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr0'], ['mac',
'00:16:3e:5c:f1:dd']]], ['device', ['vkbd']], ['device', ['vfb',
['vncunused', '1'], ['type', 'vnc'], ['xauthority', '/root/.Xauthority']]]])
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:328)
parseConfig: config is ['vm', ['name', 'guest02'], ['memory', '1024'],
['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['image',
['linux', ['kernel', '/boot/vmlinuz-2.6.18-164.6.1.el5xen'], ['ramdisk',
'/boot/initrd-2.6.18-164.6.1.el5xen.img'], ['args', 'ro']]], ['device',
['vbd', ['uname', 'phy:/dev/VMM01-KVMvg00/guest02'], ['dev', 'xvda'],
['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr0'], ['mac',
'00:16:3e:5c:f1:dd']]], ['device', ['vkbd']], ['device', ['vfb',
['vncunused', '1'], ['type', 'vnc'], ['xauthority', '/root/.Xauthority']]]]
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:445)
parseConfig: result is {'features': None, 'image': ['linux', ['kernel',
'/boot/vmlinuz-2.6.18-164.6.1.el5xen'], ['ramdisk',
'/boot/initrd-2.6.18-164.6.1.el5xen.img'], ['args', 'ro']], 'cpus': None,
'vcpu_avail': None, 'backend': [], 'uuid': None, 'on_reboot': 'restart',
'cpu_weight': None, 'memory': 1024, 'cpu_cap': None, 'localtime': None,
'timer_mode': None, 'start_time': None, 'on_poweroff': None, 'on_crash':
'restart', 'device': [('vbd', ['vbd', ['uname',
'phy:/dev/VMM01-KVMvg00/guest02'], ['dev', 'xvda'], ['mode', 'w']]), ('vif',
['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:5c:f1:dd']]), ('vkbd',
['vkbd']), ('vfb', ['vfb', ['vncunused', '1'], ['type', 'vnc'],
['xauthority', '/root/.Xauthority']])], 'bootloader': None, 'maxmem': None,
'shadow_memory': None, 'name': 'guest02', 'bootloader_args': None, 'vcpus':
1, 'cpu': None}
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1774)
XendDomainInfo.construct: None
[2009-11-06 10:27:26 xend 5112] DEBUG (balloon:145) Balloon: 3145436 KiB
free; need 4096; done.
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1900)
XendDomainInfo.initDomain: 50 256
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1941)
_initDomain:shadow_memory=0x0, maxmem=0x400, memory=0x400.
[2009-11-06 10:27:26 xend 5112] DEBUG (balloon:145) Balloon: 3145436 KiB
free; need 1048576; done.
[2009-11-06 10:27:26 xend 5112] INFO (image:139) buildDomain os=linux dom=50
vcpus=1
[2009-11-06 10:27:26 xend 5112] DEBUG (image:208) domid = 50
[2009-11-06 10:27:26 xend 5112] DEBUG (image:209) memsize = 1024
[2009-11-06 10:27:26 xend 5112] DEBUG (image:210) image =
/boot/vmlinuz-2.6.18-164.6.1.el5xen
[2009-11-06 10:27:26 xend 5112] DEBUG (image:211) store_evtchn = 1
[2009-11-06 10:27:26 xend 5112] DEBUG (image:212) console_evtchn = 2
[2009-11-06 10:27:26 xend 5112] DEBUG (image:213) cmdline = ro
[2009-11-06 10:27:26 xend 5112] DEBUG (image:214) ramdisk =
/boot/initrd-2.6.18-164.6.1.el5xen.img
[2009-11-06 10:27:26 xend 5112] DEBUG (image:215) vcpus = 1
[2009-11-06 10:27:26 xend 5112] DEBUG (image:216) features =
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController:
writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0',
'backend': '/local/domain/0/backend/vkbd/50/0'} to
/local/domain/50/device/vkbd/0.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController:
writing {'frontend-id': '50', 'domain': 'guest02', 'frontend':
'/local/domain/50/device/vkbd/0', 'state': '1', 'online': '1'} to
/local/domain/0/backend/vkbd/50/0.
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:633)
Checking for duplicate for uname: /dev/VMM01-KVMvg00/guest02
[phy:/dev/VMM01-KVMvg00/guest02], dev: xvda, mode: w
[2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device
number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-11-06 10:27:26 xend 5112] DEBUG (blkif:27) exception looking up device
number for xvda: [Errno 2] No such file or directory: '/dev/xvda'
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController:
writing {'virtual-device': '51712', 'device-type': 'disk', 'protocol':
'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vbd/50/51712'} to
/local/domain/50/device/vbd/51712.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController:
writing {'domain': 'guest02', 'frontend':
'/local/domain/50/device/vbd/51712', 'format': 'raw', 'dev': 'xvda',
'state': '1', 'params': '/dev/VMM01-KVMvg00/guest02', 'mode': 'w', 'online':
'1', 'frontend-id': '50', 'type': 'phy'} to
/local/domain/0/backend/vbd/50/51712.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController:
writing {'mac': '00:16:3e:5c:f1:dd', 'handle': '0', 'protocol':
'x86_32-abi', 'backend-id': '0', 'state': '1', 'backend':
'/local/domain/0/backend/vif/50/0'} to /local/domain/50/device/vif/0.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController:
writing {'bridge': 'xenbr0', 'domain': 'guest02', 'handle': '0', 'script':
'/etc/xen/scripts/vif-bridge', 'state': '1', 'frontend':
'/local/domain/50/device/vif/0', 'mac': '00:16:3e:5c:f1:dd', 'online': '1',
'frontend-id': '50'} to /local/domain/0/backend/vif/50/0.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:114) DevController:
writing {'protocol': 'x86_32-abi', 'state': '1', 'backend-id': '0',
'backend': '/local/domain/0/backend/vfb/50/0'} to
/local/domain/50/device/vfb/0.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:116) DevController:
writing {'vncunused': '1', 'domain': 'guest02', 'frontend':
'/local/domain/50/device/vfb/0', 'xauthority': '/root/.Xauthority', 'state':
'1', 'online': '1', 'frontend-id': '50', 'type': 'vnc'} to
/local/domain/0/backend/vfb/50/0.
[2009-11-06 10:27:26 xend 5112] DEBUG (vfbif:11) Spawn:
['/usr/lib/xen/bin/qemu-dm', '-M', 'xenpv', '-d', '50', '-domain-name',
'guest02', '-vnc', '127.0.0.1:0', '-vncunused', '-k', 'en-us']
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:992)
Storing VM details: {'shadow_memory': '0', 'uuid':
'21fa15ea-febf-1cbd-6360-cfa02eb73d8c', 'on_reboot': 'restart',
'start_time': '1257532046.49', 'on_poweroff': 'destroy', 'name': 'guest02',
'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'memory':
'1024', 'on_crash': 'restart', 'image': '(linux (kernel
/boot/vmlinuz-2.6.18-164.6.1.el5xen) (ramdisk
/boot/initrd-2.6.18-164.6.1.el5xen.img) (args ro))', 'maxmem': '1024'}
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1027)
Storing domain details: {'console/ring-ref': '1084703', 'console/port': '2',
'name': 'guest02', 'console/limit': '1048576', 'vm':
'/vm/21fa15ea-febf-1cbd-6360-cfa02eb73d8c', 'domid': '50',
'cpu/0/availability': 'online', 'memory/target': '1048576',
'store/ring-ref': '1055232', 'store/port': '1'}
[2009-11-06 10:27:26 xend.XendDomainInfo 5112] DEBUG (XendDomainInfo:1249)
XendDomainInfo.handleShutdownWatch
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices vif.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 0.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vif/50/0/hotplug-status.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vif/50/0/hotplug-status.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices usb.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices vbd.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 51712.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vbd/50/51712/hotplug-status.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices irq.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices vkbd.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 0.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vkbd/50/0/hotplug-status.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices vfb.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:164) Waiting for 0.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:509)
hotplugStatusCallback /local/domain/0/backend/vfb/50/0/hotplug-status.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:523)
hotplugStatusCallback 1.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices pci.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices ioports.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices tap.
[2009-11-06 10:27:26 xend 5112] DEBUG (DevController:158) Waiting for
devices vtpm.
[2009-11-06 10:27:26 xend 5112] INFO (XendDomain:387) Domain guest02 (50)
unpaused.
[root at VMM03-XEN xen]# xm create guest02 ; xm console guest02
Using config file "./guest02".
Started domain guest02
Linux version 2.6.18-164.6.1.el5xen (mockbuild at builder16.centos.org) (gcc
version 4.1.2 20080704 (Red Hat 4.1.2-46)) #1 SMP Tue Nov 3 17:53:47 EST
2009
BIOS-provided physical RAM map:
Xen: 0000000000000000 - 0000000040800000 (usable)
304MB HIGHMEM available.
727MB LOWMEM available.
NX (Execute Disable) protection: active
ACPI in unprivileged domain disabled
Built 1 zonelists. Total pages: 264192
Kernel command line: ro
Enabling fast FPU save and restore... done.
Enabling unmasked SIMD FPU exception support... done.
Initializing CPU#0
CPU 0 irqstacks, hard=c0750000 soft=c0730000
PID hash table entries: 4096 (order: 12, 16384 bytes)
Xen reported: 2992.688 MHz processor.
Console: colour dummy device 80x25
Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
Software IO TLB disabled
vmalloc area: ee000000-f4ffe000, maxmem 2d7fe000
Memory: 1025064k/1056768k available (2162k kernel code, 22592k reserved,
887k data, 176k init, 311304k highmem)
Checking if this processor honours the WP bit even in supervisor mode... Ok.
Calibrating delay using timer specific routine.. 7493.78 BogoMIPS
(lpj=14987562)
Security Framework v1.0.0 initialized
SELinux: Initializing.
selinux_register_security: Registering secondary module capability
Capability LSM initialized as secondary
Mount-cache hash table entries: 512
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 2048K
Checking 'hlt' instruction... OK.
SMP alternatives: switching to UP code
Freeing SMP alternatives: 13k freed
Brought up 1 CPUs
checking if image is initramfs... it is
Freeing initrd memory: 6980k freed
Grant table initialized
NET: Registered protocol family 16
Brought up 1 CPUs
PCI: setting up Xen PCI frontend stub
ACPI: Interpreter disabled.
Linux Plug and Play Support v0.97 (c) Adam Belay
pnp: PnP ACPI: disabled
xen_mem: Initialising balloon driver.
usbcore: registered new driver usbfs
usbcore: registered new driver hub
PCI: System does not support PCI
PCI: System does not support PCI
NetLabel: Initializing
NetLabel: domain hash size = 128
NetLabel: protocols = UNLABELED CIPSOv4
NetLabel: unlabeled traffic allowed by default
NET: Registered protocol family 2
IP route cache hash table entries: 32768 (order: 5, 131072 bytes)
TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
TCP bind hash table entries: 65536 (order: 7, 524288 bytes)
TCP: Hash tables configured (established 131072 bind 65536)
TCP reno registered
audit: initializing netlink socket (disabled)
type=2000 audit(1257533537.470:1): initialized
highmem bounce pool size: 64 pages
VFS: Disk quotas dquot_6.5.1
Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
Initializing Cryptographic API
alg: No test for crc32c (crc32c-generic)
ksign: Installing public key data
Loading keyring
- Added public key 5CA5E7AEC7491D45
- User ID: CentOS (Kernel Module GPG key)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
rtc: IRQ 8 is not free.
Non-volatile memory driver v1.2
Linux agpgart interface v0.101 (c) Dave Jones
brd: module loaded
Xen virtual console successfully installed as xvc0
Event-channel device installed.
Console: switching to colour frame buffer device 100x37
input: Xen Virtual Keyboard/Mouse as /class/input/input0
Uniform Multi-Platform E-IDE driver Revision: 7.00alpha2
ide: Assuming 50MHz system bus speed for PIO modes; override with idebus=xx
ide-floppy driver 0.99.newide
usbcore: registered new driver hiddev
usbcore: registered new driver usbhid
drivers/usb/input/hid-core.c: v2.6:USB HID core driver
PNP: No PS/2 controller found. Probing ports directly.
i8042.c: No controller found.
mice: PS/2 mouse device common for all mice
md: md driver 0.90.3 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bitmap version 4.39
TCP bic registered
Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI No-Shortcut mode
XENBUS: Device with no driver: device/vbd/51712
XENBUS: Device with no driver: device/vif/0
Initalizing network drop monitor service
Freeing unused kernel memory: 176k freed
Write protecting the kernel read-only data: 390k
USB Universal Host Controller Interface driver v3.0
megaraid cmm: 2.20.2.7 (Release Date: Sun Jul 16 00:01:03 EST 2006)
SCSI subsystem initialized
megaraid: 2.20.5.1 (Release Date: Thu Nov 16 15:32:35 EST 2006)
device-mapper: uevent: version 1.0.3
device-mapper: ioctl: 4.11.5-ioctl (2007-12-12) initialised:
dm-devel at redhat.com
device-mapper: dm-raid45: initialized v0.2594l
##### using virt-viewer to view the console I see one more line:
####
waiting for driver initialization...
Grovellingly yours, with bruised forehead
-Chris
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.centos.org/pipermail/centos-virt/attachments/20091106/cbec98db/attachment-0001.html
More information about the CentOS-virt
mailing list