HI,
We've started to see several issues with the Xen releases. Going back to basics I've used this guide https://wiki.centos.org/HowTos/Xen/Xen4QuickStart
Once the install process is complete the grub.conf looks like this:
default=0 timeout=5 splashimage=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title CentOS (3.18.21-17.el6.x86_64) root (hd0,0) kernel /boot/xen.gz dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all module /boot/vmlinuz-3.18.21-17.el6.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet title CentOS (2.6.32-573.12.1.el6.x86_64) root (hd0,0) kernel /boot/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /boot/initramfs-2.6.32-573.12.1.el6.x86_64.img title CentOS (2.6.32-358.el6.x86_64) root (hd0,0) kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=71b203ab-36ea-4aa8-9ba3-78e4109f0ca4 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /boot/initramfs-2.6.32-358.el6.x86_64.img
As you can see the missing module line. If i manually run grub-bootxen.sh nothing changes.
Manually adding the missing module entry:
module /boot/initramfs-3.18.21-17.el6.x86_64.img
Causes the following:
https://www.dropbox.com/s/ley5nj29ubwqogc/Screenshot%202015-12-20%2011.10.11...
Disabling APIC gives the following console output:
https://www.dropbox.com/s/a6zgioxl9xpg20y/Screenshot%202015-12-20%2011.15.29...
Now i know this is inside a KVM virtual machine, however we are starting to see this issue on at least 7 standard dedicated servers.
[root@test ~]# rpm -qa | grep xen xen-libs-4.4.3-9.el6.x86_64 xen-hypervisor-4.4.3-9.el6.x86_64 xen-4.4.3-9.el6.x86_64 centos-release-xen-7-11.el6.x86_64 xen-licenses-4.4.3-9.el6.x86_64 xen-runtime-4.4.3-9.el6.x86_64
[root@test ~]# cat /etc/redhat-release CentOS release 6.7 (Final) [root@test ~]#
[root@test ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz stepping : 10 microcode : 1 cpu MHz : 2666.352 cache size : 4096 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon unfair_spinlock pni ssse3 cx16 sse4_1 x2apic xsave hypervisor lahf_lm bogomips : 5332.70
I've also had a few Xen systems grub entry issues. I'm not yet sure what triggers the events, but I do think there *may* be a correlation to the "/boot" part. I've found the Xen servers that have "/boot" before any of the the lines like "/boot/xen.gz" ended up missing complete grub entries. Not sure it's 100% true. Anyone else comment?
Thanks PJ
On Sun, Dec 20, 2015 at 5:31 AM, Phill Bandelow phill@onapp.com wrote:
HI,
We've started to see several issues with the Xen releases. Going back to basics I've used this guide https://wiki.centos.org/HowTos/Xen/Xen4QuickStart
Once the install process is complete the grub.conf looks like this:
default=0 timeout=5 splashimage=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title CentOS (3.18.21-17.el6.x86_64) root (hd0,0) kernel /boot/xen.gz dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all module /boot/vmlinuz-3.18.21-17.el6.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet title CentOS (2.6.32-573.12.1.el6.x86_64) root (hd0,0) kernel /boot/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /boot/initramfs-2.6.32-573.12.1.el6.x86_64.img title CentOS (2.6.32-358.el6.x86_64) root (hd0,0) kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=71b203ab-36ea-4aa8-9ba3-78e4109f0ca4 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /boot/initramfs-2.6.32-358.el6.x86_64.img
As you can see the missing module line. If i manually run grub-bootxen.sh nothing changes.
Manually adding the missing module entry:
module /boot/initramfs-3.18.21-17.el6.x86_64.img
Causes the following:
https://www.dropbox.com/s/ley5nj29ubwqogc/Screenshot%202015-12-20%2011.10.11...
Disabling APIC gives the following console output:
https://www.dropbox.com/s/a6zgioxl9xpg20y/Screenshot%202015-12-20%2011.15.29...
Now i know this is inside a KVM virtual machine, however we are starting to see this issue on at least 7 standard dedicated servers.
[root@test ~]# rpm -qa | grep xen xen-libs-4.4.3-9.el6.x86_64 xen-hypervisor-4.4.3-9.el6.x86_64 xen-4.4.3-9.el6.x86_64 centos-release-xen-7-11.el6.x86_64 xen-licenses-4.4.3-9.el6.x86_64 xen-runtime-4.4.3-9.el6.x86_64
[root@test ~]# cat /etc/redhat-release CentOS release 6.7 (Final) [root@test ~]#
[root@test ~]# cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 23 model name : Intel(R) Core(TM)2 Quad CPU Q9400 @ 2.66GHz stepping : 10 microcode : 1 cpu MHz : 2666.352 cache size : 4096 KB physical id : 0 siblings : 1 core id : 0 cpu cores : 1 apicid : 0 initial apicid : 0 fpu : yes fpu_exception : yes cpuid level : 13 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx lm constant_tsc arch_perfmon unfair_spinlock pni ssse3 cx16 sse4_1 x2apic xsave hypervisor lahf_lm bogomips : 5332.70
-- Regards, Phill
CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
On Sun, Dec 20, 2015 at 5:11 PM, PJ Welsh pjwelsh@gmail.com wrote:
I've also had a few Xen systems grub entry issues. I'm not yet sure what triggers the events, but I do think there *may* be a correlation to the "/boot" part. I've found the Xen servers that have "/boot" before any of the the lines like "/boot/xen.gz" ended up missing complete grub entries. Not sure it's 100% true. Anyone else comment?
I can't quite figure out what you mean by having a "/boot" before other lines. Could you attach copies of "normal" vs "buggy" grub.conf files?
-George
On Sun, Dec 20, 2015 at 11:31 AM, Phill Bandelow phill@onapp.com wrote:
HI,
We've started to see several issues with the Xen releases. Going back to basics I've used this guide https://wiki.centos.org/HowTos/Xen/Xen4QuickStart
Once the install process is complete the grub.conf looks like this:
default=0 timeout=5 splashimage=(hd0,0)/boot/grub/splash.xpm.gz hiddenmenu title CentOS (3.18.21-17.el6.x86_64) root (hd0,0) kernel /boot/xen.gz dom0_mem=1024M,max:1024M cpuinfo com1=115200,8n1 console=com1,tty loglvl=all guest_loglvl=all module /boot/vmlinuz-3.18.21-17.el6.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet title CentOS (2.6.32-573.12.1.el6.x86_64) root (hd0,0) kernel /boot/vmlinuz-2.6.32-573.12.1.el6.x86_64 ro root=/dev/vda1 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /boot/initramfs-2.6.32-573.12.1.el6.x86_64.img title CentOS (2.6.32-358.el6.x86_64) root (hd0,0) kernel /boot/vmlinuz-2.6.32-358.el6.x86_64 ro root=UUID=71b203ab-36ea-4aa8-9ba3-78e4109f0ca4 rd_NO_LUKS rd_NO_LVM LANG=en_US.UTF-8 rd_NO_MD SYSFONT=latarcyrheb-sun16 crashkernel=auto KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet initrd /boot/initramfs-2.6.32-358.el6.x86_64.img
As you can see the missing module line. If i manually run grub-bootxen.sh nothing changes.
Thanks for the detailed report.
The initrd thing is a bug in the upstream script that makes the grub config; I haven't seen the behavior, so I haven't been able to track down what the problem is, but other people have[1].
CentOS 7 uses grub2, which comes with its own grub-generation code which is more reliable.
[1] marc.info/?i=563670F4.9030701@karlos.cz
Manually adding the missing module entry:
module /boot/initramfs-3.18.21-17.el6.x86_64.img
Causes the following:
https://www.dropbox.com/s/ley5nj29ubwqogc/Screenshot%202015-12-20%2011.10.11...
Disabling APIC gives the following console output:
https://www.dropbox.com/s/a6zgioxl9xpg20y/Screenshot%202015-12-20%2011.15.29...
Now i know this is inside a KVM virtual machine, however we are starting to see this issue on at least 7 standard dedicated servers.
You're starting to see that exact error message, you mean, about the timer not connected to the IOAPIC?
In any case, it would be more helpful if you could manage to get a serial console output from one of the physical boxes where you've got a problem. A lot of servers have built-in serial consoles you can attach to over the network; otherwise, you can take a look at this page:
http://wiki.xen.org/wiki/Xen_Serial_Console
Thanks, -George
-
George Dunlap -
Phill Bandelow -
PJ Welsh