[CentOS] Commands failing silently?

Mon Mar 24 21:19:00 UTC 2008
Dan Bongert <dbongert at wisc.edu>

mouss wrote:
> Dan Bongert wrote:
>> Hello all:
>>
>> I have a couple CentOS 4 servers (all up-to-date) that are having 
>> strange command failures. I first noticed this with a perl script that 
>> uses lots of system calls.
>>
>> thoth(66) /tmp> uname -a
>> Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 06:54:55 
>> EDT 2008 i686 i686 i386 GNU/Linux
>>
>> Nothing in either dmesg or /var/log/messages seems to indicate any 
>> problems. It also doesn't seem to matter what the command is -- ls is 
>> the quickest test, but sshd will sometimes to fail to spawn children, 
>> etc. There aren't a large amount of processes on the machine either -- 
>> only 122 at the moment.
>>
>> Has anyone seen this behavior before? Have I been hit with some sort 
>> of cunning rootkit? This machine shouldn't be publicly accessible; 
>> it's behind our firewall.
> 
> where is /tmp mounted? is this an external disk (usb, ...)? is it an nfs 
> mount?

It's a local disk:

thoth(97) /tmp> df -h .
Filesystem            Size  Used Avail Use% Mounted on
/dev/md4               16G   77M   15G   1% /tmp

Though 'ls' was just an example -- just about any program will fail. The 'w'
command will fail too:

thoth(118) /tmp> w
   16:06:51 up  5:34,  1 user,  load average: 0.94, 1.46, 2.04
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
dbongert pts/0    copland.ssc.wisc 14:16    0.00s  0.22s  0.05s w

thoth(119) /tmp> w
   16:06:52 up  5:34,  1 user,  load average: 0.94, 1.46, 2.04
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
dbongert pts/0    copland.ssc.wisc 14:16    0.00s  0.22s  0.05s w

thoth(120) /tmp> w

thoth(121) /tmp> w

-- 
Dan Bongert                     dbongert at wisc.edu