Hello Everyone,
I have been trying to get a Fedora 12 domU to boot for the better part of the afternoon and haven't had success booting any F12 kernel in Xen.
I can get Fedora 11 installed no problem, tried to do a preupgrade but the when booting the preupgrade kernel through grub (or even from outside the domU) the domU reboots very quickly and ultimately crashes.
As far as I saw in the qemu-dm-xxx.log, xend.log, and xend-debug.log there are no messages informing me of what went wrong.
I am using CentOS 5.3 with the 2.6.18-92.1.22.el5xen kernel. I am unable to update the kernel at this time as new drivers cause major disk controller issues.
If it as simple as a kernel update I will try again, but as of right now I am stumped so I asking for a little help!
Has anyone successfully gotten F12 set as a domU using the regular CentOS xen and kernel-xen packages?
xen-3.0.3-80.el5_3.3 kernel-xen-2.6.18-92.1.22.el5
Best, Tait
On Wed, Jan 06, 2010 at 06:11:12PM -0500, Tait Clarridge wrote:
I have been trying to get a Fedora 12 domU to boot for the better part of the afternoon and haven't had success booting any F12 kernel in Xen.
I have not been able to get F12 i386 domU to run on CentOS 5.4 x86_64 dom0, but the x86_64 F12 domU is running fine.
On Wed, Jan 06, 2010 at 06:11:12PM -0500, Tait Clarridge wrote:
Hello Everyone,
I have been trying to get a Fedora 12 domU to boot for the better part of the afternoon and haven't had success booting any F12 kernel in Xen.
I can get Fedora 11 installed no problem, tried to do a preupgrade but the when booting the preupgrade kernel through grub (or even from outside the domU) the domU reboots very quickly and ultimately crashes.
As far as I saw in the qemu-dm-xxx.log, xend.log, and xend-debug.log there are no messages informing me of what went wrong.
I am using CentOS 5.3 with the 2.6.18-92.1.22.el5xen kernel. I am unable to update the kernel at this time as new drivers cause major disk controller issues.
I think 5.4 added some hypervisor/dom0 side fixes, that might be needed for F12 guests. I'm not totally sure. You can go through redhat bugzillas if you want :)
If it as simple as a kernel update I will try again, but as of right now I am stumped so I asking for a little help!
Has anyone successfully gotten F12 set as a domU using the regular CentOS xen and kernel-xen packages?
xen-3.0.3-80.el5_3.3 kernel-xen-2.6.18-92.1.22.el5
I've been successfully running both 32bit and 64bit Fedora 12 domUs/guests on EL 5.4 dom0.
What's the actual problem? Use virt-install or virt-manager to install new F12 guest.
Btw what's your hardware?
-- Pasi
I think 5.4 added some hypervisor/dom0 side fixes, that might be needed for F12 guests. I'm not totally sure. You can go through redhat bugzillas if you want :)
I will take a look into the changes, thanks.
I've been successfully running both 32bit and 64bit Fedora 12 domUs/guests on EL 5.4 dom0.
What's the actual problem? Use virt-install or virt-manager to install new F12 guest.
Btw what's your hardware?
-- Pasi
To install the guest I mounted the DVD on a remote webserver and exported via HTTP, from dom0 I grabbed the pxeboot kernel and ramdisk and passed those to the domU to start the install. When using the F12 kernel/ramdisk the domU doesn't cause any errors (logs are clean as well) but it doesn't start. It goes up and down twice in very quick succession and the logs tell me that it is rebooting too quickly and therefore it stops attempting to boot the domU. When using the F11 install kernel/ramdisk the domU boots and installs just fine.
To get it to F12 I tried preupgrade but on rebooting when I select the preupgrade selection from pygrub it has the same behaviour as the initial F12 install. Same thing happens when I do the upgrade from yum and try to boot the new kernel.
The machine right now is running all F12 software except for the kernel. When booting any F12 kernel there are no error messages in either domU (obviously) or dom0.
The hardware is a SunFire x4140 with 64GB ram and dual quad core Opterons. I can get more specific if you would like, but xen has worked perfectly on these machines with every other type of OS except F12.
I will try and get some downtime for the box so I can upgrade if the conclusion is that some dom0/hypervisor fixes in the newer kernel/xen packages will solve the problem.
I appreciate the help and quick responses so far :)
Tait
On Thu, Jan 07, 2010 at 09:43:32AM -0500, Tait Clarridge wrote:
I think 5.4 added some hypervisor/dom0 side fixes, that might be needed for F12 guests. I'm not totally sure. You can go through redhat bugzillas if you want :)
I will take a look into the changes, thanks.
I've been successfully running both 32bit and 64bit Fedora 12 domUs/guests on EL 5.4 dom0.
What's the actual problem? Use virt-install or virt-manager to install new F12 guest.
Btw what's your hardware?
-- Pasi
To install the guest I mounted the DVD on a remote webserver and exported via HTTP, from dom0 I grabbed the pxeboot kernel and ramdisk and passed those to the domU to start the install. When using the F12 kernel/ramdisk the domU doesn't cause any errors (logs are clean as well) but it doesn't start. It goes up and down twice in very quick succession and the logs tell me that it is rebooting too quickly and therefore it stops attempting to boot the domU. When using the F11 install kernel/ramdisk the domU boots and installs just fine.
To get it to F12 I tried preupgrade but on rebooting when I select the preupgrade selection from pygrub it has the same behaviour as the initial F12 install. Same thing happens when I do the upgrade from yum and try to boot the new kernel.
The machine right now is running all F12 software except for the kernel. When booting any F12 kernel there are no error messages in either domU (obviously) or dom0.
The hardware is a SunFire x4140 with 64GB ram and dual quad core Opterons. I can get more specific if you would like, but xen has worked perfectly on these machines with every other type of OS except F12.
I will try and get some downtime for the box so I can upgrade if the conclusion is that some dom0/hypervisor fixes in the newer kernel/xen packages will solve the problem.
I appreciate the help and quick responses so far :)
I think 5.4 version of python-virtinst added some fixes aswell.. so maybe you need to upgrade to that for F12 installation to work.
Also please try running "xm console" for the guest and see what it does and where it crashes..
-- Pasi
I think 5.4 version of python-virtinst added some fixes aswell.. so maybe you need to upgrade to that for F12 installation to work.
Also please try running "xm console" for the guest and see what it does and where it crashes..
If I do xm create -c f12domU.cfg it says "Starting Domain f12domU" and then brings me back to the command prompt on dom0.
Thanks for the replies Pasi, I will try and get some time to bring this host out of production to perform the upgrade.
- Tait
On Fri, 2010-01-08 at 09:38 -0500, Tait Clarridge wrote:
I think 5.4 version of python-virtinst added some fixes aswell.. so maybe you need to upgrade to that for F12 installation to work.
Also please try running "xm console" for the guest and see what it does and where it crashes..
If I do xm create -c f12domU.cfg it says "Starting Domain f12domU" and then brings me back to the command prompt on dom0.
Just do add something, this led me to believe there was an incompatible kernel, and have tried both i686 and x86_64 kernels.
Thanks for the replies Pasi, I will try and get some time to bring this host out of production to perform the upgrade.
- Tait
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
On Fri, Jan 08, 2010 at 10:08:25AM -0500, Tait Clarridge wrote:
On Fri, 2010-01-08 at 09:38 -0500, Tait Clarridge wrote:
I think 5.4 version of python-virtinst added some fixes aswell.. so maybe you need to upgrade to that for F12 installation to work.
Also please try running "xm console" for the guest and see what it does and where it crashes..
If I do xm create -c f12domU.cfg it says "Starting Domain f12domU" and then brings me back to the command prompt on dom0.
Just do add something, this led me to believe there was an incompatible kernel, and have tried both i686 and x86_64 kernels.
Make sure your domU kernel has these kernel cmdline options: "console=hvc0 earlyprintk=xen"
That makes the Xen console work with upstream pv_ops kernels.
Also, if you think it crashes, you can add "on_crash=preserve" option to the Xen guest configfile, and when the guest has crashed you can do:
/usr/lib/xen/bin/xenctx -s System.map-domUkernelversion <domid>
That'll give you a stacktrace of the crashes guest to debug it. Note that if your dom0 is 64bit the xenctx binary is under lib64, not lib.
Also, what hardware was this? There has been some fixes to the incorrect cpuid masking causing guest kernel crashes on new cpus, and some of those are fixed in upcoming EL 5.5 Xen.
-- Pasi
On Sat, Jan 09, 2010 at 03:45:15PM +0200, Pasi Kärkkäinen wrote:
On Fri, Jan 08, 2010 at 10:08:25AM -0500, Tait Clarridge wrote:
Also please try running "xm console" for the guest and see what it does and where it crashes..
Make sure your domU kernel has these kernel cmdline options: "console=hvc0 earlyprintk=xen"
Do these options work with the F11 guest kernel (2.6.30) as well?
Also, what hardware was this? There has been some fixes to the incorrect cpuid masking causing guest kernel crashes on new cpus, and some of those are fixed in upcoming EL 5.5 Xen.
Are these going to be in 5.4?
I'm running 2.6.18-164.11.1.el5xen and the fedora kernels for F11 and F12 are not very stable for me. Dom0 is a Centos 5.4 2.6.18-164.9.1.el5xen kernel.
These fedora kernels seem to lockup (processes get stuck in D state) whenever put under any load:
kernel-2.6.30.10-105.fc11.x86_64 kernel-2.6.31.6-166.fc12.x86_64 kernel-2.6.31.9-174.fc12.x86_64
Yes, I tried a fedora 11 kernel on a fedora 12 system. This is the most stable for me.
Here are some bugzilla entries (one posted by me) that point to this problem:
kernel 2.6.31 processes lock up in D state https://bugzilla.redhat.com/show_bug.cgi?id=550724
FC12 2.6.31.9-174.fc12.x86_64 hangs under heavy disk I/O https://bugzilla.redhat.com/show_bug.cgi?id=551552
Any help appreciated.
On Sat, Jan 30, 2010 at 10:57:02AM +1100, Norman Gaywood wrote:
On Sat, Jan 09, 2010 at 03:45:15PM +0200, Pasi Kärkkäinen wrote:
On Fri, Jan 08, 2010 at 10:08:25AM -0500, Tait Clarridge wrote:
Also please try running "xm console" for the guest and see what it does and where it crashes..
Make sure your domU kernel has these kernel cmdline options: "console=hvc0 earlyprintk=xen"
Do these options work with the F11 guest kernel (2.6.30) as well?
Yes, they'll work for all pv_ops kernels.
Also, what hardware was this? There has been some fixes to the incorrect cpuid masking causing guest kernel crashes on new cpus, and some of those are fixed in upcoming EL 5.5 Xen.
Are these going to be in 5.4?
Not sure. At least in 5.5.
I'm running 2.6.18-164.11.1.el5xen and the fedora kernels for F11 and F12 are not very stable for me. Dom0 is a Centos 5.4 2.6.18-164.9.1.el5xen kernel.
These fedora kernels seem to lockup (processes get stuck in D state) whenever put under any load:
kernel-2.6.30.10-105.fc11.x86_64 kernel-2.6.31.6-166.fc12.x86_64 kernel-2.6.31.9-174.fc12.x86_64
Yes, I tried a fedora 11 kernel on a fedora 12 system. This is the most stable for me.
Do you have some easy-to-reproduce test/script so I could try it on my system?
Here are some bugzilla entries (one posted by me) that point to this problem:
kernel 2.6.31 processes lock up in D state https://bugzilla.redhat.com/show_bug.cgi?id=550724
FC12 2.6.31.9-174.fc12.x86_64 hangs under heavy disk I/O https://bugzilla.redhat.com/show_bug.cgi?id=551552
-- Pasi
On Mon, Feb 01, 2010 at 10:20:26AM +0200, Pasi Kärkkäinen wrote:
On Sat, Jan 30, 2010 at 10:57:02AM +1100, Norman Gaywood wrote:
On Sat, Jan 09, 2010 at 03:45:15PM +0200, Pasi Kärkkäinen wrote:
On Fri, Jan 08, 2010 at 10:08:25AM -0500, Tait Clarridge wrote:
Also please try running "xm console" for the guest and see what it does and where it crashes..
Make sure your domU kernel has these kernel cmdline options: "console=hvc0 earlyprintk=xen"
Do these options work with the F11 guest kernel (2.6.30) as well?
Yes, they'll work for all pv_ops kernels.
Also, what hardware was this? There has been some fixes to the incorrect cpuid masking causing guest kernel crashes on new cpus, and some of those are fixed in upcoming EL 5.5 Xen.
Are these going to be in 5.4?
Not sure. At least in 5.5.
I'm running 2.6.18-164.11.1.el5xen and the fedora kernels for F11 and F12 are not very stable for me. Dom0 is a Centos 5.4 2.6.18-164.9.1.el5xen kernel.
These fedora kernels seem to lockup (processes get stuck in D state) whenever put under any load:
kernel-2.6.30.10-105.fc11.x86_64 kernel-2.6.31.6-166.fc12.x86_64 kernel-2.6.31.9-174.fc12.x86_64
Yes, I tried a fedora 11 kernel on a fedora 12 system. This is the most stable for me.
Do you have some easy-to-reproduce test/script so I could try it on my system?
Here are some bugzilla entries (one posted by me) that point to this problem:
kernel 2.6.31 processes lock up in D state https://bugzilla.redhat.com/show_bug.cgi?id=550724
FC12 2.6.31.9-174.fc12.x86_64 hangs under heavy disk I/O https://bugzilla.redhat.com/show_bug.cgi?id=551552
I haven't seen these D state problems, but I found this CONFIG_HIGHPTE xen_set_pte() bug/race that causes 32bit PAE guest crashes:
https://bugzilla.redhat.com/show_bug.cgi?id=566932
Were your guests 32bit?
-- Pasi
On Sat, Feb 20, 2010 at 05:09:31PM +0200, Pasi Kärkkäinen wrote:
On Mon, Feb 01, 2010 at 10:20:26AM +0200, Pasi Kärkkäinen wrote:
On Sat, Jan 30, 2010 at 10:57:02AM +1100, Norman Gaywood wrote:
On Sat, Jan 09, 2010 at 03:45:15PM +0200, Pasi Kärkkäinen wrote:
I'm running 2.6.18-164.11.1.el5xen and the fedora kernels for F11 and F12 are not very stable for me. Dom0 is a Centos 5.4 2.6.18-164.9.1.el5xen kernel.
These fedora kernels seem to lockup (processes get stuck in D state) whenever put under any load:
kernel-2.6.30.10-105.fc11.x86_64 kernel-2.6.31.6-166.fc12.x86_64 kernel-2.6.31.9-174.fc12.x86_64
Do you have some easy-to-reproduce test/script so I could try it on my system?
No unfortuately. It's very difficult to pin down the workload that causes this. The test systems I've tried don't seem to trigger it. The crash does not even seem to be related to a heavy load. We will have several LTSP users and a little bit of disk activity and suddenly it will trip into the D state lockup. This could be after many hours (up to 2 days) of running, or in our worst case 2 minutes.
Here are some bugzilla entries (one posted by me) that point to this problem:
kernel 2.6.31 processes lock up in D state https://bugzilla.redhat.com/show_bug.cgi?id=550724
FC12 2.6.31.9-174.fc12.x86_64 hangs under heavy disk I/O https://bugzilla.redhat.com/show_bug.cgi?id=551552
And this one as well seems to be the same issue if you look at the end of the report:
F12 Xen DomU unstable (2.6.31) https://bugzilla.redhat.com/show_bug.cgi?id=526627
I haven't seen these D state problems, but I found this CONFIG_HIGHPTE xen_set_pte() bug/race that causes 32bit PAE guest crashes:
https://bugzilla.redhat.com/show_bug.cgi?id=566932
Were your guests 32bit?
No, they are 64bit.
If I google around, I can find people which look like they have had similar issues. Most threads die out with no resolution. Here is one that looks similar on the kernel mailing list:
http://www.mail-archive.com/kvm@vger.kernel.org/msg23039.html
-
Neil Thompson -
Norman Gaywood -
Pasi Kärkkäinen -
Tait Clarridge