google said that is a bug of kernel in network driver.

And, someone seems to present a patch to fix the bug, however I would like to know whether it can work from others' works becuause reinstalling a new kernel may let us cost a lot.

On Sun, Jul 27, 2008 at 12:45 AM, Johnny Hughes johnny@centos.org wrote:

Ian jonhson wrote:

...

Dear All,

I selected CentOS5 in my works and installed them in two DELL PowerEdge1950. However, a trouble blocked me during the machines run after two days. The machines crashed and the syslog said it got the following messages:

------------------ part dump of /var/log/messages ----------------------

..... Jul 25 02:15:02 vega2008 kernel: [<c045bc58>] ? exit_mmap+0x93/0xc9 Jul 25 02:15:02 vega2008 kernel: [<c04214c2>] ? mmput+0x25/0x68 Jul 25 02:15:02 vega2008 kernel: [<c046e9c9>] ? flush_old_exec+0x4f8/0x777 Jul 25 02:15:02 vega2008 kernel: [<c046dfcf>] ? kernel_read+0x32/0x43 Jul 25 02:15:02 vega2008 kernel: [<c0490e60>] ? load_elf_binary+0x359/0x1152 Jul 25 02:15:02 vega2008 kernel: [<c045a6ee>] ? get_user_pages+0x2d5/0x35c Jul 25 02:15:02 vega2008 kernel: [<c04570d2>] ? page_address+0x78/0x98 Jul 25 02:15:02 vega2008 kernel: [<c045735a>] ? kmap_high+0x19/0x16b Jul 25 02:15:02 vega2008 kernel: [<c04570d2>] ? page_address+0x78/0x98 Jul 25 02:15:02 vega2008 kernel: [<c046dced>] ? copy_strings+0x169/0x173 Jul 25 02:15:02 vega2008 kernel: [<c046ddad>] ? search_binary_handler+0x8f/0x1af Jul 25 02:15:02 vega2008 kernel: [<c046efe7>] ? do_execve+0x133/0x194 Jul 25 02:15:02 vega2008 kernel: [<c04030d7>] ? sys_execve+0x2a/0x4a Jul 25 02:15:02 vega2008 kernel: [<c04047aa>] ? syscall_call+0x7/0xb Jul 25 02:15:02 vega2008 kernel: [<c0610000>] ? early_init_intel+0x0/0x3c Jul 25 02:15:02 vega2008 kernel: ======================= Jul 25 02:15:02 vega2008 kernel: BUG: soft lockup - CPU#7 stuck for 61s! [sshd:24188] Jul 25 02:15:02 vega2008 kernel: Jul 25 02:15:02 vega2008 kernel: Pid: 24188, comm: sshd Not tainted (2.6.25.3 #3) Jul 25 02:15:02 vega2008 kernel: EIP: 0060:[<c06154f0>] EFLAGS: 00200293 CPU: 7 Jul 25 02:15:02 vega2008 kernel: EIP is at _spin_lock+0xa/0x15 Jul 25 02:15:02 vega2008 kernel: EAX: c079349c EBX: f79ec580 ECX: ffffffff EDX: 00008381 Jul 25 02:15:02 vega2008 kernel: ESI: ffffffff EDI: f79ec580 EBP: f68a6580 ESP: f1824e50 Jul 25 02:15:02 vega2008 kernel: DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068 Jul 25 02:15:02 vega2008 kernel: CR0: 8005003b CR2: b7ebf978 CR3: 32996000 CR4: 000006f0 Jul 25 02:15:02 vega2008 kernel: DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 Jul 25 02:15:02 vega2008 kernel: DR6: ffff0ff0 DR7: 00000400 Jul 25 02:15:02 vega2008 kernel: [<c04119c7>] ? native_flush_tlb_others+0x49/0x9b Jul 25 02:15:02 vega2008 kernel: [<c0411e65>] ? flush_tlb_mm+0x51/0x54 Jul 25 02:15:02 vega2008 kernel: [<c045bc58>] ? exit_mmap+0x93/0xc9 Jul 25 02:15:02 vega2008 kernel: [<c04214c2>] ? mmput+0x25/0x68 Jul 25 02:15:02 vega2008 kernel: [<c046e9c9>] ? flush_old_exec+0x4f8/0x777 Jul 25 02:15:02 vega2008 kernel: [<c046dfcf>] ? kernel_read+0x32/0x43 Jul 25 02:15:02 vega2008 kernel: [<c0490e60>] ? load_elf_binary+0x359/0x1152 Jul 25 02:15:02 vega2008 kernel: [<c045a6ee>] ? get_user_pages+0x2d5/0x35c Jul 25 02:15:02 vega2008 kernel: [<c04570d2>] ? page_address+0x78/0x98 Jul 25 02:15:02 vega2008 kernel: [<c045735a>] ? kmap_high+0x19/0x16b Jul 25 02:15:02 vega2008 kernel: [<c04570d2>] ? page_address+0x78/0x98 Jul 25 02:15:02 vega2008 kernel: [<c046dced>] ? copy_strings+0x169/0x173 Jul 25 02:15:02 vega2008 kernel: [<c046ddad>] ? search_binary_handler+0x8f/0x1af Jul 25 02:15:02 vega2008 kernel: [<c046efe7>] ? do_execve+0x133/0x194 Jul 25 02:15:02 vega2008 kernel: [<c04030d7>] ? sys_execve+0x2a/0x4a Jul 25 02:15:02 vega2008 kernel: [<c04047aa>] ? syscall_call+0x7/0xb Jul 25 02:15:02 vega2008 kernel: [<c0610000>] ? early_init_intel+0x0/0x3c Jul 25 02:15:02 vega2008 kernel: ======================= Jul 25 02:15:02 vega2008 kernel: BUG: soft lockup - CPU#1 stuck for 61s! [http_cap:12228] .............

---

I choose CentOS because I believe it is the most stable OS in commodity machines. But I don't know how to do when facing the trouble.

The difference lies in that I patched a PF_RING patch in original kernel and recompiled the kernel to run my machines. I wonder whether the patched kernel crashes the CentOS because PF_RING automatically downloads the kernel codes from www.kernel.org, but not the one from www.centos.org? Or, the centOS holds the bugs in its distribution?

This is very common, a google search for:

'poweredge 1950' 'BUG: soft lockup' 'stuck'

produces almost 2000 results. It seems to be something to do with the on board network ports.

I do not seen this problem ... has anyone else?

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Ian jonhson wrote:

BTW, the patched kernel by PF_RING is version 2.6.25.3.

I do not have anything that new in testing .. however we do have this, which you might try to see if it still happens:

http://people.centos.org/hughesjr/kernel-rt/

Dell might also have modules posted for this machine.

These say EL5 should work:

https://hardware.redhat.com/show.cgi?id=227919

https://hardware.redhat.com/show.cgi?id=375801

When you use a custom kernel, you usually get to keep the pieces.

Go buy a new car and change the engine and then see if they honor the warranty.

We stopped using Dell machines some time ago. For some reason there reliability has been steadily falling. I don't know if they are doing non-standard stuff in their bios's or just using less than optimal hardware to cut costs, but I probably won't go back.

agree. I have turn back to original CentOS kernel, and then restart my program. In other words, my program run without PF_RING. By now, nothing happens. Therefore, it seems the bug results from PF_RING patch.

Thanks for your help.

Johnny Hughes wrote:

We don't have too many servers (about 20 in 5 locations) or workstations (about 150 in 5 locations) but I have had no problem with Dell machines or their service ... maybe I am just lucky.

The company I'm at has about 380 dell servers and probably 100 desktops. For the most part they work fine.

We did have a problem with firmware on Seagate drives that came in the servers though. The company spent months trying to track down the source of the problem before I was hired. It didn't take me too long to narrow it down. The issue was multithreaded reads/writes to the disk under their application was 10x slower on some systems than others. Normal disk benchmark tools didn't pick up anything unusual. And my co-workers said Dell support was worthless for anything other than flat out RMA (I've never dealt with them so can't say from personal experience). My experience with HP support has been similar though, so I don't doubt it.

But I noticed pretty quick after I was hired that the systems that had problems had Seagate drives, and the ones that did not had Maxtor, or Fujitsu drives. Unfortunately upgrading the drive firmware was a painful procedure involving booting to DOS. After updating the problem was gone, of course.

Why some drives shipped with Maxtor, some with Fujitsu, and some with Seagate drives I don't know(all the same model#). I've been told that Dell has a history of swapping out components to whichever is the cheapest that week, it seems that's the case at least with HDDs.

Never personally had a problem with firmware on a disk in my experience with thousands of drives over the past 15 years or so. Not sure how that got past Dell's "QA". Maybe it was a compatibility issue with their RAID controller and that firmware rev on the disk.

I suppose the main thing I don't like about Dell that I did like about the HP systems was being able to monitor the RAID array was pretty simple with HP, we installed a tool called 'hpacucli', no extra drivers needed, and it worked fine. With Dell the only tool I've found is 'raidcfg', and that seems to require a bunch of extra packages and drivers to be installed. And on at least a couple different types of Dell systems it causes them to hang when I run it. I really don't like installing "extra" drivers for management stuff. I never installed the HP management packs, and don't plan to touch the Dell stuff either(short of raidcfg and it's dependencies).

Dell doesn't seem to be too bad though, I honestly expected more problems given the pricing of the systems. Some of the pricing is even cheaper than a local Supermicro reseller, and I'm talking low quantities even. One of my former employers switched to Dell after they were bought out by a bigger company(bigger company's policy) and they pay at least 50% less than we do.

I've only been working with them for a few months now.

nate

On Tue, Jul 29, 2008 at 4:44 AM, Johnny Hughes johnny@centos.org wrote:

well ... my $work has been using dell exclusively for servers and workstations for 5 years.

I have had no real issues with either the poweredge servers or the desktops and Dell has always sent parts in a timely manner to replace any broken system under warranty.

We don't have too many servers (about 20 in 5 locations) or workstations (about 150 in 5 locations) but I have had no problem with Dell machines or their service ... maybe I am just lucky.

Nope - blessed.

:-)

mhr