<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=utf-8">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7652.24">
<TITLE>Re: [CentOS-devel] RE: [CentOS-virt] BUG: soft lockup detected onCPU#?</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<BR>
<P><FONT SIZE=2>Thanks Johnny, and sorry for the top post (blackberry).<BR>
<BR>
I d/l'd the src rpm and found their patches all in 1 patch file called xen.patch (I did an ls -lt and picked the files with the latest timestamps). There may also be kernel config changes as several config files were touched, but I couldn't get a hold of the original 8.1.8 src rpm to diff them.<BR>
<BR>
I would be happy to help in getting the parts needed so they can be rolled up into a single patch to apply to the current plus kernel. Just let me know what you need.<BR>
<BR>
I wonder if anybody at XenSource notified upstream of the fixes?<BR>
<BR>
-Ross<BR>
<BR>
<BR>
----- Original Message -----<BR>
From: centos-devel-bounces@centos.org <centos-devel-bounces@centos.org><BR>
To: The CentOS developers mailing list. <centos-devel@centos.org><BR>
Sent: Wed Jan 23 07:37:04 2008<BR>
Subject: Re: [CentOS-devel] RE: [CentOS-virt] BUG: soft lockup detected onCPU#?<BR>
<BR>
Ross S. W. Walker wrote:<BR>
> Ross S. W. Walker wrote:<BR>
>> Brett Worth wrote:<BR>
>>> Hello All.<BR>
>>><BR>
>>> I've just started looking into Xen and have a test<BR>
>>> environment in place. I'm seeing an<BR>
>>> annoying problem that I thought worthy of a post.<BR>
>>><BR>
>>> Config:<BR>
>>><BR>
>>> I have 2 x HP DL585 servers each with 4 Dual core Opterons<BR>
>>> (non-vmx) and 16GB RAM<BR>
>>> configured as Xen servers. These run CentOS 5.1 with the<BR>
>>> latest updates applied. These<BR>
>>> system both attach to an iSCSI target which is an HP DL385<BR>
>>> running ietd and serving SAN<BR>
>>> based storage.<BR>
>>><BR>
>>> I have a test VM running CentOS 5.1 also updated.<BR>
>>><BR>
>>> Problem:<BR>
>>><BR>
>>> If I run the VM on a single server everything is OK. If I do<BR>
>>> a migrate of the VM to the<BR>
>>> other server I start getting random "BUG: soft lockup<BR>
>>> detected on CPU#?" messages on the<BR>
>>> VM console. The messages seem to happen with IO but not<BR>
>>> every time. A reboot of the VM<BR>
>>> on the new server will stop these messages.<BR>
>>><BR>
>>> I've also left the VM running overnight a couple of times and<BR>
>>> when I do I find that any<BR>
>>> external sessions (ssh) are hung in the morning but the<BR>
>>> console session is not. New ssh<BR>
>>> sessions can be started and seem to work.<BR>
>>><BR>
>>> After much googling it looks like the kernel messages can<BR>
>>> occur if dom0 is very busy but<BR>
>>> mine is not.<BR>
>>><BR>
>>> Any suggestions?<BR>
>> The soft lockup is technically not a BUG.<BR>
>><BR>
>> You will see these errors if an IRQ takes more then 10 seconds<BR>
>> to respond.<BR>
>><BR>
>> In your case I would take a look at your iSCSI setup and the<BR>
>> time it takes to migrate the VM from one node to another along<BR>
>> with SCSI reserve/release setup on the iSCSI target.<BR>
>><BR>
>> I also have been using the Xen 3.2 RPMs off xen.org to CentOS<BR>
>> 5.1 which good results, the VM migration may run smoother and<BR>
>> quicker in Xen 3.2, but in doing so you take Xen off the<BR>
>> reservation, if your OK with that it may fix your issues.<BR>
><BR>
> After seeing this same issue on my Xen 3.2 install, but with NO<BR>
> migration or iSCSI happening I decided it is probably NOT iSCSI's<BR>
> fault, so I decided to research it a little more and this is what<BR>
> I found:<BR>
><BR>
> <A HREF="http://docs.xensource.com/XenServer/4.0.1/guest/ch04s08.html#rhel5_limitations">http://docs.xensource.com/XenServer/4.0.1/guest/ch04s08.html#rhel5_limitations</A><BR>
><BR>
> XenSource does provide a repo of CentOS 5 kernels that have been<BR>
> patched to fix this though:<BR>
><BR>
> <A HREF="http://updates.xensource.com/XenServer/4.0.1/centos5x/">http://updates.xensource.com/XenServer/4.0.1/centos5x/</A><BR>
><BR>
> But these seem to be woefully out of date.<BR>
><BR>
> I wonder if a kind soul would add the fix to the centosplus kernel<BR>
> with XenSource's patch so those rogue Xen users could benefit from<BR>
> this fix until upstream decides to include it.<BR>
><BR>
> I suppose the centosplus patch would need to be flagged interm in<BR>
> case it needs removed when upstream has their own fix.<BR>
<BR>
Ross,<BR>
<BR>
Thanks for researching this.<BR>
<BR>
I can probably add this to the next centosplus kernels, though I usually<BR>
do not like to add patches ... and I will need to grab their kernels and<BR>
work out what is patched and try to roll it into our kernels.<BR>
<BR>
-- Johnny Hughes<BR>
<BR>
</FONT>
</P>
<P></P>
<HR WIDTH="100%">
This e-mail, and any attachments thereto, is intended only for use by
the addressee(s) named herein and may contain legally privileged
and/or confidential information. If you are not the intended recipient
of this e-mail, you are hereby notified that any dissemination,
distribution or copying of this e-mail, and any attachments thereto,
is strictly prohibited. If you have received this e-mail in error,
please immediately notify the sender and permanently delete the
original and any copy or printout thereof.
</BODY>
</HTML>