[CentOS] Commands failing silently?

Mon Mar 24 19:27:47 UTC 2008
Dan Bongert <dbongert at wisc.edu>

Hello all:

I have a couple CentOS 4 servers (all up-to-date) that are having strange 
command failures. I first noticed this with a perl script that uses lots of 
system calls.

Basically, sometimes a command just won't run:

thoth(52) /tmp> ls

thoth(53) /tmp> ls

thoth(54) /tmp> ls

thoth(55) /tmp> ls
learner  lost+found/

thoth(56) /tmp> ls
learner  lost+found/

thoth(57) /tmp> ls
learner  lost+found/

thoth(58) /tmp> ls
learner  lost+found/

thoth(59) /tmp> ls
learner  lost+found/

thoth(60) /tmp> ls
learner  lost+found/

thoth(61) /tmp> ls
learner  lost+found/

thoth(62) /tmp> ls

thoth(63) /tmp> ls

thoth(64) /tmp> ls

thoth(65) /tmp> ls

thoth(66) /tmp> uname -a
Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 EDT 
2008 i686 i686 i386 GNU/Linux

Nothing in either dmesg or /var/log/messages seems to indicate any problems. 
It also doesn't seem to matter what the command is -- ls is the quickest 
test, but sshd will sometimes to fail to spawn children, etc. There aren't a 
large amount of processes on the machine either -- only 122 at the moment.

Has anyone seen this behavior before? Have I been hit with some sort of 
cunning rootkit? This machine shouldn't be publicly accessible; it's behind 
our firewall.

Thanks.
-- 
Dan Bongert                     dbongert at wisc.edu