[CentOS] Commands failing silently?

[mouss]

Dan Bongert wrote:
> Hello all:
>
> I have a couple CentOS 4 servers (all up-to-date) that are having 
> strange command failures. I first noticed this with a perl script that 
> uses lots of system calls.
>
> Basically, sometimes a command just won't run:
>
> thoth(52) /tmp> ls
>
> thoth(53) /tmp> ls
>
> thoth(54) /tmp> ls
>
> thoth(55) /tmp> ls
> learner  lost+found/
>
> thoth(56) /tmp> ls
> learner  lost+found/
>
> thoth(57) /tmp> ls
> learner  lost+found/
>
> thoth(58) /tmp> ls
> learner  lost+found/
>
> thoth(59) /tmp> ls
> learner  lost+found/
>
> thoth(60) /tmp> ls
> learner  lost+found/
>
> thoth(61) /tmp> ls
> learner  lost+found/
>
> thoth(62) /tmp> ls
>
> thoth(63) /tmp> ls
>
> thoth(64) /tmp> ls
>
> thoth(65) /tmp> ls
>
> thoth(66) /tmp> uname -a
> Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 
> EDT 2008 i686 i686 i386 GNU/Linux
>
> Nothing in either dmesg or /var/log/messages seems to indicate any 
> problems. It also doesn't seem to matter what the command is -- ls is 
> the quickest test, but sshd will sometimes to fail to spawn children, 
> etc. There aren't a large amount of processes on the machine either -- 
> only 122 at the moment.
>
> Has anyone seen this behavior before? Have I been hit with some sort 
> of cunning rootkit? This machine shouldn't be publicly accessible; 
> it's behind our firewall.

where is /tmp mounted? is this an external disk (usb, ...)? is it an nfs 
mount?