[CentOS] How to tell if I've been hacked?

Dave tdbtdb+centos at gmail.com
Sat Aug 22 08:37:49 UTC 2009

On Tue, Aug 18, 2009 at 3:53 PM, Scott Ehrlich<srehrlich at gmail.com> wrote:
> There is a lot of talk about the vulnerable Linux kernel.   I'm simply
> wondering the telltale signs if a given system has been hacked?
> What, specifically, does a person look for?

This is an interesting and frustrating question. Perfect security is
impossible, but maybe we can achieve 'good enough'.

On Tue, Aug 18, 2009 at 5:14 PM, Christopher
Chan<christopher.chan at bradbury.edu.hk> wrote:
> rpm -Va is a good start for modified binaries/libraries.

Problems with rpm: 1) Many files on a system did not come from an rpm
or had good reasons to change, so dealing with false positives is a
problem 2) Some packages are sloppy and no longer verify immediately
after installation 3) rpm cannot address memory-only attacks or bios
attacks. Still, if rpm tells you some binary has changed since
installation, you know you're in trouble. Or you're using prelink.

> rootkit detectors is another thing you can try.

Problem with rootkit detectors I have used: many false positives. At
least, I hope they were false! Googling around, I found that once you
had a positive result, it was sort of complicated process to figure
out whether you'd really been hacked or were just having timing
problems or had an unusual configuration.

> Other than that, it is checking your logs and looking for odd files
> lying around...

And prayer.

On Tue, Aug 18, 2009 at 5:22 PM, Ryan Pugatch<rpug at tripadvisor.com> wrote:
> Also, processes running that you don't recognize.

Unfortunately, if you're like me there are really a lot of processes
running on a virgin linux box (never touched the internet) that I
don't recognize. I once tried just making a big file of them and
having a cron job send me email when the list changed significantly.
This could have caught an unlucky or inept cracker who launched some
process named "meEvilCrackhead", but wouldn't have done much to catch
someone using an innocuous name, like say 'grep'.

>Users you don't
> recognize.

Again, it is possible to catch someone who doesn't bother to get rid
of the smoking gun. Someone who has root on your system can create a
new user, or they could use a pre-existing user. You can keep an eye
out for strange users, but the real problem is spotting familiar users
doing stuff they ought not. Even that can be covered if the cracker
replaces your tools or hacks the kernel.

> Logged in sessions that you don't recognize.

I'm not sure what Ryan means here, unless he is assuming only one
person (you) has authorized access to your machine, and you see
sessions logged in as you that you know nothing about. Yeah, that
would tip you off. If lots of users can log in, there's not much

> Free space
> shrinking abnormally.

Again not really sure what this would mean. Too high a load, too many
programs running? Again, someone with root access could hack the tools
you use to monitor this, or even the kernel, and make it really hard
to see. Assuming you really know how much free space you ought to have
at a given moment, which, for me, I am ashamed to admit, would be
quite rare.

> An increase in bandwidth usage that is unexpected.

Now we're talking! Well, I am still pretty damn ignorant of what a
system's bandwidth demand ought to be, but at least you could see the
stuff actually happening and make a sort of reasonable investigation
of 'what do I have running that would possibly want to talk to IP
xxx.yyy.zzz.aaa?'  And for once, no matter how good the intruder is,
they won't be able to get your own system to lie to you for them
(assuming you're using a different system to do the network analysis).
But while you analize the traffic, the bad guys has more time to
damage your data.

On Tue, Aug 18, 2009 at 5:58 PM, Christopher
Chan<christopher.chan at bradbury.edu.hk> wrote:
> Yeah...one should not assume that those will be hidden by rogue
> libraries/binaries. Not every case will be taken that far or unspotted
> before it gets that far.

Every intrusion is vulnerable for a while at least, while the intruder
is trying to get in and get root. After that they will probably try to
cover their tracks.

On Tue, Aug 18, 2009 at 6:57 PM, Bill Campbell<centos at celestial.com> wrote:
> To really know whether a system has been hacked, it's necessary
> to use something like Tripwire or Aide,

And very carefully. Only that won't help you with memory-only attacks,
or bios stuff, etc. These tools concentrate on verifying that your
disk files have not been altered. I don't think they would help with
an attack that uses free space (guessing here). Also, they are a pain,
unless your system stays absolutely static, which in effect means, if
you never use it. Have them ignore your data space, and the hacker can
exploit that. And even then, linux is constantly updating various
files in the background, and of course you need to update software to
keep up with the security patches. You need to track every change of
every file. I doubt many people have the patience.


> It's also a good idea to check for executables in places they
> normally shouldn't be, /tmp, /dev/shm on SuSE systems, /var/tmp,
> and similar directories where crackers like to hide their work.
> Often these executes will be in directories with names like ``.. ''
> (note the trailing space) that look legitimate.

I like this, because it might actually be automated. Of course, you're
trusting stat or whatever.


> You cannot trust tools like ``ps'', ``find'', ``netstat'', and
> ``lsof'' as these are frequently replaced by ones that are
> modified to hide the cracker's work.

Naturally we are running aide and tripwire from a CD or other
read-only medium, why not toss in a copy of these tools as well? Of
course, if the kernel has been hacked, even that won't save us, but we
have to take what we can get.

On Wed, Aug 19, 2009 at 1:03 AM, Eduardo
Grosclaude<eduardo.grosclaude at gmail.com> wrote:
> As a corollary, the only safe way to audit a suspected system is
> booting your diagnostic tool from known good media (eg try a security
> Live CD distro)

Safe in what sense? When you rebooted, you may have erased from memory
the only remaining trace of the intrusion and you are still
vulnerable. But at least you *can* trust what the tools are telling
you - I hope.

On Thu, Aug 20, 2009 at 12:59 PM, Magnus
Holmström<magnus.holmstrom at gmail.com> wrote:

Oops, someone is going to scold you for top-posting.

> Check for failed logins in /var/log/messages

Kind of useless, don't you think?  Failed logins I can live with. It's
successful ones that hurt.

> Check if the /etc/passwd file have been changed

Certainly, no decent detection system should overlook changes to
/etc/passwd. But that's just one of many important files.

> Use commands like last, w and uptime.

These tools are kind of weak. If you have a single-user system or the
intruder is doing something really obvious (what? rm -rf /?  wall 'I
am an intruder!'?) they would help. With multiple users they won't
tell you enough, and if the intruder is good they will lie to you.

Thanks to all for stimulating thoughts. I've been thinking about this
for a while. I need to think some more.

More information about the CentOS mailing list