On one of my systems I seem to loose a file or two from time to time. Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was deleted/removed/vanished. Another time it was /home/online/sh/daemon that was deleted.
But I can't seem to find anything strange in the logs or in the history, nor would any of my scripts running in crontab mess with those files.
Where can I look for clues? And how do I enable audit for file operations in my home folder?
/Christopher Thorjussen
On Thu, 3 Jan 2008 13:09:11 +0100 "Christopher Thorjussen" Christopher.Thorjussen@carrot.no wrote:
On one of my systems I seem to loose a file or two from time to time.
Where can I look for clues?
Is your system visible to the internet? Maybe it's running some kind of Apache with homedirs loosely enabled and one unsecure php script + one little h4x0r could do the trick? Is your file system sane? Is your hard drive(s) SMART-wise OK? CM
On Thu, 3 Jan 2008 13:09:11 +0100 "Christopher Thorjussen" Christopher.Thorjussen@carrot.no wrote:
On one of my systems I seem to loose a file or two from time to time.
Where can I look for clues?
Is your system visible to the internet? Maybe it's running some kind of Apache with homedirs loosely enabled and one unsecure php script + one little h4x0r could do the trick? Is your file system sane? Is your
hard
drive(s) SMART-wise OK? CM
The system is visible only for a few defined IP addresses on the internet
Apache is not installed. It runs Oracle 10.2g
/Christopher
On Thu, 2008-01-03 at 13:09 +0100, Christopher Thorjussen wrote:
On one of my systems I seem to loose a file or two from time to time. Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was deleted/removed/vanished. Another time it was /home/online/sh/daemon that was deleted.
But I can't seem to find anything strange in the logs or in the history, nor would any of my scripts running in crontab mess with those files.
Where can I look for clues? And how do I enable audit for file operations in my home folder?
Although less common than "In The Day", if file system corruption was repaired by *fsck during boot or other times, the "lost+found" directory of the affected FS may contain some files with weird names, having mostly numbers. These numbers are i-node numbers associated with the "chunks" that appear in the lost+found directory.
If you see these, it indicates that corruption occurred and was repaired. If the files are plain text, there is a possibility to reconstruct the original files via visual inspection and concatenation. Often they are mixed or binary and more difficult to reconstruct.
If it's not corruption, finding the cause may not be so easy. Long ago, before the we had the plethora of tools now available, I found the cause of such a mystery with a multi-pronged approach.
1. Fire up the system accounting package (sar). 2. Start a "daemon" (really just a "nohup <your shell script name> &" that checked for the existence of the file every so many seconds. This if really nothing more than a
if [ ! -r <file name> ] ; then <some commands> ; fi
The "-r" can be replaced/compounded with other flags as desired. See "man bash", "CONDITIONAL EXPRESSIONS.
When the file went missing, a "log" entry was generated.
3. Generate the sar reports and examine them to see what was running around that time.
4. As I did, go viciously beat the offending user that made a flawed script about the ears with a rather rigid clue bat.
5. Problem solved.
/Christopher Thorjussen
<snip sig stuff>
HTH
On one of my systems I seem to loose a file or two from time to time. Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was deleted/removed/vanished. Another time it was /home/online/sh/daemon that was deleted.
But I can't seem to find anything strange in the logs or in the history, nor would any of my scripts running in crontab mess with those files.
Where can I look for clues? And how do I enable audit for file operations in my home folder?
If your system is capable, use the SMART tools to check your drive out (as CM suggests), something like this:
smartctl -a /dev/sda <replace /dev/sda with the drive in question> See how your 'error count log' is doing. If there are errors, then you might want to run that command a few times and see if the error count is still rising.
Is it everything in the /home/online/sh/ directory that is getting deleted, or can you see any pattern at all? (it sounds like it is random from what you said...but hard to think of why files would be deleted randomly....as you know!)
Where can I look for clues? And how do I enable audit for file operations in my home folder?
If your system is capable, use the SMART tools to check your drive out (as CM suggests), something like this:
smartctl -a /dev/sda <replace /dev/sda with the drive in question> See how your 'error count log' is doing. If there are errors, then you might want to run that command a few times and see if the error count
is
still rising.
It's a Dell PowerEdge 2950 running in raid 1+0 on the Perc 5/I with SCSI drives.
[root@ora01 ~]# smartctl -a /dev/sda7 smartctl version 5.33 [x86_64-redhat-linux-gnu] Copyright (C) 2002-4 Bruce Allen Home page is http://smartmontools.sourceforge.net/
Device: DELL PERC 5/i Version: 1.03 Serial number: 008f71137876e77c0e00b4fdc230c201 Device type: disk Local Time is: Fri Jan 4 09:43:37 2008 CET Device does not support SMART
Error Counter logging not supported
Error Events logging not supported Device does not support Self Test logging
Is it everything in the /home/online/sh/ directory that is getting deleted, or can you see any pattern at all? (it sounds like it is
random
from what you said...but hard to think of why files would be deleted randomly....as you know!)
No pattern so far. Yeah I know it sounds strange for files to be randomly deleted.
/Christopher
On Thursday 03 January 2008 19:09:11 Christopher Thorjussen wrote:
On one of my systems I seem to loose a file or two from time to time. Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was deleted/removed/vanished. Another time it was /home/online/sh/daemon that was deleted.
But I can't seem to find anything strange in the logs or in the history, nor would any of my scripts running in crontab mess with those files.
Where can I look for clues? And how do I enable audit for file operations in my home folder?
Hi, this really sounds weird. In order to audit it, the following checklist might help: 1. If the system was administered by an admin other than you and he got fired/dismissed with hard feeling on him, he might put a crontab that would do nasty thing randomly. Audit all the files in: /var/spool/cron /var/spool/at Also all the script in /etc/cron.{d,daily,weekly,monthly}, /etc/crontab
2, Audit all RPM files installed using: rpm -Va, looks for a difference in md5sum for binary files such as /bin/ls,/bin/ps, etc. You might want to use cracker detection script such as rkhunter.
3. Looks for the word "error" in log files: grep -r error /var/log See for related error such as filesystem corruption, etc
4. It's a long shot, but could be a misconfigured rsync script?
HTH, pls let us know the result.
On Thursday 03 January 2008 19:09:11 Christopher Thorjussen wrote:
On one of my systems I seem to loose a file or two from time to
time.
Last night, one of my files (/home/online/sh/NattjobbPrivat.sh) was deleted/removed/vanished. Another time it was /home/online/sh/daemon that was deleted.
But I can't seem to find anything strange in the logs or in the
history,
nor would any of my scripts running in crontab mess with those
files.
Where can I look for clues? And how do I enable audit for file operations in my home folder?
Hi, this really sounds weird. In order to audit it, the following checklist might help:
- If the system was administered by an admin other than you and he
got
fired/dismissed with hard feeling on him, he might put a crontab that would do nasty thing randomly. Audit all the files in: /var/spool/cron /var/spool/at Also all the script in /etc/cron.{d,daily,weekly,monthly},
/etc/crontab
No admin or anyone else with access have quit or been fired. The files and folders looks fine.
2, Audit all RPM files installed using: rpm -Va, looks for a difference in md5sum for binary files such as /bin/ls,/bin/ps, etc. You might want to use cracker detection
script
such as rkhunter.
The files look fine. Some files are marked as MD5 mismatch but it's mostly config files I've changed. The only files I'm not sure of is:
SM5....T /usr/share/rhn/rhn_applet/rhn_applet.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_animation.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_apt.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_dialogs.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_model.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_protocols.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_rpc.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_rpm.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_version.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_applet_yum.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_sources.pyc SM5....T /usr/share/rhn/rhn_applet/rhn_utils.pyc
But I'm not running X so the applet isn't running.
- Looks for the word "error" in log files:
grep -r error /var/log See for related error such as filesystem corruption, etc
[root@ora01 tmp]# grep -r error /var/log /var/log/Xorg.0.log: (WW) warning, (EE) error, (NI) not implemented, (??) unknown. /var/log/anaconda.log:* getting rpm error class /var/log/prelink.log:/usr/lib64/libgpg-error.so.0.1.3 0000003c50e00000-0000003c50f02878 /var/log/rpmpkgs.4:libgpg-error-1.0-1.x86_64.rpm /var/log/rpmpkgs.1:libgpg-error-1.0-1.x86_64.rpm /var/log/messages.2:Dec 17 08:13:10 ora01 kernel: daemon[1562]: segfault at 0000007fc0000000 rip 0000002a957af4b2 rsp 0000007fbfffe730 error 6 /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/scrollkeeper.log:I/O error : Attempt to load network entity http://scrollkeeper.sourceforge.net/dtds/scrollkeeper-omf-1.0/scrollkeep er-omf.dtd /var/log/rpmpkgs.2:libgpg-error-1.0-1.x86_64.rpm /var/log/Xorg.0.log.old: (WW) warning, (EE) error, (NI) not implemented, (??) unknown. /var/log/rpmpkgs.3:libgpg-error-1.0-1.x86_64.rpm /var/log/rpmpkgs:libgpg-error-1.0-1.x86_64.rpm /var/log/anaconda.xlog: (WW) warning, (EE) error, (NI) not implemented, (??) unknown. /var/log/anaconda.xlog:error opening security policy file /etc/X11/xserver/SecurityPolicy
- It's a long shot, but could be a misconfigured rsync script?
Rsync is not running/used, but some custom scripts are running cleaning up some folders. I'm trying to battle through them to see if somethings wrong in them, but so far I've found nothing.
HTH, pls let us know the result.
Will do.
/Christopher
-
Christopher Thorjussen -
CM -
Fajar Priyanto -
mike.redan＠bell.ca -
William L. Maltby