[CentOS] CentOS 4.4-IBM Netvista Performace Problems, help needed.

Sun Feb 18 13:52:32 UTC 2007
Chuck Mattern <camattern at acm.org>

I've got an odd situation that I need some advise on.  I have two 
computers that I am planning to use as a cluster.  I initially started 
with some left over Compaq Presairos with 667MHz CPUs.  I loaded CentOS 
4.3 and later updated to 4.4.  Things ran normally, albeit slowly.  I 
had an opportunity to upgrade to a pair of IBM Netvistas with 2.26 GHz 
CPUs, I did this by transferring the 160GB Western Digital IDE disks and 
NICs but did not re-install the OS, just migrated the disks.  Since then 
they have had the following symptoms:

-Systems frequently boot faster than the disks can be spun up and have 
to be soft booted to recognize and boot from the disks.

-Systems will bog down critically after approximately 24 hours loosing 
system time at an increasing rate, ofrr instance a for loop that runs 
date, hwclock ans then sleeps for 10 minutes will show time in sync for 
the first few hours then the system time will begin to fall behind at an 
increasing rate, after 24 hours the system time essentially stops 
elapsing.  It almost feels like the box has trouble processing 
interrupts.  Once it gets to this state performance becomes very 
sluggish, for instance top will take up to 90 seconds to display it's 
first screen and will not update on it's own, only when Enter is 
depressed.  At times top will show 0's across the utilization line for 
everything including idle.  I have gone as far as to boot one of the 
boxes into single user mode and run the date/hwclock loop and even in 
that state the system will bog down and gradually stop elapsing time 
after 18-24 hours.   Even shutting down is impacted.  A reboot will take 
well over an hour to process.

I had a copy of Ubuntu on my desk and have booted into that distro from 
cd and it passes the date/hwclock test (actually lost 2 seconds over a 
24 hour period but I can live with that via ntp).  I'm downloading a 
copy of the CentOS live 4.4 cd and will try this with that as well but 
at this time does anyone see that this could be something other than a 
disk incompatibility with the newer systems?  Should I try 
re-installing?  If it is the disks, any thoughts on something I could 
try to avoid buying new disks (I have tried setting the BIOS to both the 
high performance and legacy disk modes (not entirely sure what's behind 
that IBMism)).

Regards,
Chuck