On Wed, 2011-04-13 at 13:06 -0700, Florin Andrei wrote: > Running v5 64bit on a Dell 1950. > > A cluster of 3 DB machines, identical hardware. One of them suddenly > became slower 2 weeks ago. > > tar -zxf with a large file on this machine takes 1.5 minutes, but takes > only 10 seconds on any of its siblings. CPU usage seems high while > untarring, with lots of user and sys cycles being used, but almost no > wait cycles. It doesn't matter whether I untar on a local disk, or on a > fiber channel SAN volume, it's slow anyway. > > scp a file over the network is slow too: 6 MB/s to this machine, 70 MB/s > to its siblings. > > However, this is just as fast on all systems, including the "sick" one: > > # time dd if=/dev/zero of=/dev/null bs=1M count=100000 > 100000+0 records in > 100000+0 records out > 104857600000 bytes (105 GB) copied, 2.59213 seconds, 40.5 GB/s > > real 0m2.600s > user 0m0.025s > sys 0m2.550s > > /proc/cpuinfo looks fine. Nothing suspect in dmesg. > > Reboot doesn't fix it. Power off / power on doesn't fix it. Single mode > is slow too, and I tried a couple different kernels. > > Dell's online diagnostics program could find nothing wrong with it. > > /var/log/messages was full of "ntpd[7313]: frequency error -1707 PPM > exceeds tolerance 500 PPM" messages. There was a lot of messages about > "the system limit for the maximum number of semaphore sets has been > exceeded"; there was indeed a lot of leftover semaphores created by NRPE > (owned by the nagios user); I deleted them but nothing has changed, so > they were a symptom, not the cause. Are the system times different between the siblings? Are all 3 siblings running ntpd and using the same time source (server(s))? Do the symptoms change with ntpd stopped/running? Are the frequency offsets the same on each sibling? Since your log messages appear to be ntp related, you might try resetting your frequency offset and drift values. Having a -1707 PPM offset could cause many issues like you describe. service ntpd stop ntptime -f 0 echo "0" > /var/lib/ntp/drift service ntpd start > I'm still kind of hoping it's a software issue, but chances are slim. > OTOH, I can't imagine any hardware problem that would exhibit these > symptoms. > > Any idea what to test? >