[CentOS] Commands failing silently?

Thu Mar 27 15:30:20 UTC 2008
Dan Bongert <dbongert at wisc.edu>

mouss wrote:
> Dan Bongert wrote:
>> mouss wrote:
>>> Dan Bongert wrote:
>>>> Hello all:
>>>>
>>>> I have a couple CentOS 4 servers (all up-to-date) that are having 
>>>> strange command failures. I first noticed this with a perl script 
>>>> that uses lots of system calls.
>>>>
>>>> thoth(66) /tmp> uname -a
>>>> Linux thoth.ssc.wisc.edu 2.6.9-67.0.7.ELsmp #1 SMP Sat Mar 15 
>>>> 06:54:55 EDT 2008 i686 i686 i386 GNU/Linux
>>>>
>>>> Nothing in either dmesg or /var/log/messages seems to indicate any 
>>>> problems. It also doesn't seem to matter what the command is -- ls 
>>>> is the quickest test, but sshd will sometimes to fail to spawn 
>>>> children, etc. There aren't a large amount of processes on the 
>>>> machine either -- only 122 at the moment.
>>>>
>>>> Has anyone seen this behavior before? Have I been hit with some sort 
>>>> of cunning rootkit? This machine shouldn't be publicly accessible; 
>>>> it's behind our firewall.
>>>
>>> where is /tmp mounted? is this an external disk (usb, ...)? is it an 
>>> nfs mount?
>>
>> It's a local disk:
>>
>> thoth(97) /tmp> df -h .
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/md4               16G   77M   15G   1% /tmp
>>
>> Though 'ls' was just an example -- just about any program will fail. 
>> The 'w'
>> command will fail too:
> 
> 
> maybe check your PATH. try
> $ /bin/ls

Ok, here's a heck of a thing. When I run 'ls' using the full path (and also 
when I unalias it -- I have 'ls' aliased to 'ls -F --color'), 'ls' no longer 
fails.

However, my other test case, 'w', still fails.

(and these are all test cases because I noticed a nightly job with a lot of 
system() calls was failing).


-- 
Dan Bongert                     dbongert at wisc.edu