Guys, apache cpus usage is hitting 100% sometimes ( to such an extent that its very noticeable) on a box with just 8 users or so.
i m getting this when i run 'top'. The worrying thing is seeing the work 'atack' under command
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack 22375 apache 15 0 964 560 472 S 0.3 0.0 0:04.21 atack 22858 apache 15 0 964 560 472 S 0.3 0.0 0:02.87 atack 22997 apache 15 0 964 560 472 S 0.3 0.0 0:04.11 atack 22999 apache 15 0 964 560 472 S 0.3 0.0 0:02.22 atack 23007 apache 15 0 964 560 472 S 0.3 0.0 0:03.79 atack 23099 apache 15 0 964 556 472 S 0.3 0.0 0:02.18 atack 23101 apache 15 0 964 556 472 S 0.3 0.0 0:02.48 atack 23108 apache 15 0 964 556 472 S 0.3 0.0 0:03.59 atack 23109 apache 15 0 964 556 472 S 0.3 0.0 0:02.75 atack 23112 apache 15 0 972 504 412 S 0.3 0.0 0:04.70 atack 23115 apache 15 0 964 556 472 S 0.3 0.0 0:03.75 atack 23116 apache 15 0 964 556 472 S 0.3 0.0 0:02.80 atack 23121 apache 15 0 972 504 412 S 0.3 0.0 0:03.79 atack 23384 apache 15 0 964 556 472 S 0.3 0.0 0:01.63 atack 23389 apache 15 0 964 556 472 S 0.3 0.0 0:03.52 atack 23392 apache 15 0 964 556 472 S 0.3 0.0 0:01.61 atack 23397 apache 15 0 964 556 472 S 0.3 0.0 0:01.62 atack 23405 apache 15 0 964 556 472 S 0.3 0.0 0:03.64 atack
When i 'ps -ef' i can see many lines as below;
apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100 apache 24292 23378 0 11:00 ? 00:00:01 ./atack 100 apache 24335 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24344 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24347 23378 0 11:02 ? 00:00:00 ./atack 100 apache 24358 23378 0 11:04 ? 00:00:00 ./atack 100
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
sorry typos amended....
Guys, apache's cpu usage is hitting 100% sometimes ( to such an extent that its very noticeable) on a box ( 2gb ram) with just 8 users or so. This newver happended before.
i m getting this when i run 'top'. The worrying thing is seeing the word 'atack' under command
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack 22375 apache 15 0 964 560 472 S 0.3 0.0 0:04.21 atack 22858 apache 15 0 964 560 472 S 0.3 0.0 0:02.87 atack 22997 apache 15 0 964 560 472 S 0.3 0.0 0:04.11 atack 22999 apache 15 0 964 560 472 S 0.3 0.0 0:02.22 atack 23007 apache 15 0 964 560 472 S 0.3 0.0 0:03.79 atack 23099 apache 15 0 964 556 472 S 0.3 0.0 0:02.18 atack 23101 apache 15 0 964 556 472 S 0.3 0.0 0:02.48 atack 23108 apache 15 0 964 556 472 S 0.3 0.0 0:03.59 atack 23109 apache 15 0 964 556 472 S 0.3 0.0 0:02.75 atack 23112 apache 15 0 972 504 412 S 0.3 0.0 0:04.70 atack 23115 apache 15 0 964 556 472 S 0.3 0.0 0:03.75 atack 23116 apache 15 0 964 556 472 S 0.3 0.0 0:02.80 atack 23121 apache 15 0 972 504 412 S 0.3 0.0 0:03.79 atack 23384 apache 15 0 964 556 472 S 0.3 0.0 0:01.63 atack 23389 apache 15 0 964 556 472 S 0.3 0.0 0:03.52 atack 23392 apache 15 0 964 556 472 S 0.3 0.0 0:01.61 atack 23397 apache 15 0 964 556 472 S 0.3 0.0 0:01.62 atack 23405 apache 15 0 964 556 472 S 0.3 0.0 0:03.64 atack
When i 'ps -ef' i can see many lines as below;
apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100 apache 24292 23378 0 11:00 ? 00:00:01 ./atack 100 apache 24335 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24344 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24347 23378 0 11:02 ? 00:00:00 ./atack 100 apache 24358 23378 0 11:04 ? 00:00:00 ./atack 100
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
On Tue, Jun 02, 2009 at 08:23:16PM -0700, Linux Advocate wrote:
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
Yes. Reinstall; fully update components; restore *data* from backups (you have backups, right?) and review what web packages you have installed and make sure those are fully updated also.
Your box is compromised. You have no way to gauge the severity, so treat it as both a lost cause; nothing on it can be trusted at this point.
John
John R. Dennison wrote:
On Tue, Jun 02, 2009 at 08:23:16PM -0700, Linux Advocate wrote:
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
Yes. Reinstall; fully update components; restore *data* from backups (you have backups, right?) and review what web packages you have installed and make sure those are fully updated also.
Your box is compromised. You have no way to gauge the severity, so treat it as both a lost cause; nothing on it can be trusted at this point.
John
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
some google foo shows this is a WINDOWS exploit not a linux one.
http://www.linuxquestions.org/questions/slackware-14/analyzing-apache-logs-1...
some google foo shows this is a WINDOWS exploit not a linux one.
http://www.linuxquestions.org/questions/slackware-14/analyzing-apache-logs-1... _______________________________________________
yes, william, i saw those links when i googled....i too did no think it related to me bcos i am on a centos box...
On Tue, Jun 02, 2009 at 11:48:11PM -0400, William Warren wrote:
some google foo shows this is a WINDOWS exploit not a linux one.
http://www.linuxquestions.org/questions/slackware-14/analyzing-apache-logs-1...
Um, perhaps I am just missing something but I don't see any relation of that forum thread (dating from '03 I might add) to the issue that the original poster has.
John
John R. Dennison wrote:
On Tue, Jun 02, 2009 at 08:23:16PM -0700, Linux Advocate wrote:
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
Yes. Reinstall; fully update components; restore *data* from backups (you have backups, right?) and review what web packages you have installed and make sure those are fully updated also.
Your box is compromised. You have no way to gauge the severity, so treat it as both a lost cause; nothing on it can be trusted at this point.
John
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
http://www.derkeiler.com/Newsgroups/comp.os.linux.security/2004-05/0202.html
William Warren wrote:
http://www.derkeiler.com/Newsgroups/comp.os.linux.security/2004-05/0202.html
This has nothing to do with the issue at hand (neither did the other URL from your earlier mail).
It can *clearly* be seen that there are processes running as the apache user on that box - so why do you link to URLs explaining *LOG ENTRIES* pertaining to some obscure windows bug from 5 years ago?
Ralph
reply below
----- Original Message ----
From: John R. Dennison jrd@gerdesas.com To: CentOS mailing list centos@centos.org Sent: Wednesday, June 3, 2009 11:43:46 AM Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
On Tue, Jun 02, 2009 at 08:23:16PM -0700, Linux Advocate wrote:
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
Yes. Reinstall; fully update components; restore *data* from backups (you have backups, right?) and review what web packages you have installed and make sure those are fully updated also. Your box is compromised. You have no way to gauge the severity, so treat it as both a lost cause; nothing on it can be trusted at this point.
ohhhhhhhhhhhhhhhh .... godddddddddddddd.....................
i have a quite a few linux boxes and not even one has been hacked..... oh man !!!!!!
really??? i have to format the box.....
i have a quite a few linux boxes and not even one has been hacked..... oh man !!!!!!
Consider yourself lucky that you have not had it happen in the past. Nothing is 100% secure.
really??? i have to format the box.....
Yes, you do.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
On Tue, Jun 02, 2009 at 09:01:35PM -0700, Linux Advocate wrote:
ohhhhhhhhhhhhhhhh .... godddddddddddddd.....................
i have a quite a few linux boxes and not even one has been hacked..... oh man !!!!!!
That you have noticed.
really??? i have to format the box.....
Yes, it would be extremely irresponsible for you to allow that box to remain connected to the 'net. It's been compromised and as such it's a rogue server.
John
on 6-2-2009 9:09 PM John R. Dennison spake the following:
On Tue, Jun 02, 2009 at 09:01:35PM -0700, Linux Advocate wrote:
ohhhhhhhhhhhhhhhh .... godddddddddddddd.....................
i have a quite a few linux boxes and not even one has been hacked..... oh man !!!!!!
That you have noticed.
really??? i have to format the box.....
Yes, it would be extremely irresponsible for you to allow that box to remain connected to the 'net. It's been compromised and as such it's a rogue server.
And if you have other server set up identically, you might want to check/secure them before they too are "owned"
And if you have other server set up identically, you might want to check/secure them before they too are "owned"
Nevermind identically; you should check all of your systems. If this is a business environment, you should really think about getting a professional vulnerability assessment or at least a tool to do a vulnerability assessment that you can run yourself.
-geoff
--------------------------------- Geoff Galitz Blankenheim NRW, Germany http://www.galitz.org/ http://german-way.com/blog/
Hello:
If there are processes running on your machine which you do not recognize, assume the machine has been compromised. Take it offline and wipe it immediately.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Linux Advocate Sent: Tuesday, June 02, 2009 10:23 PM To: CentOS mailing list Subject: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
Guys, apache cpus usage is hitting 100% sometimes ( to such an extent that its very noticeable) on a box with just 8 users or so.
i m getting this when i run 'top'. The worrying thing is seeing the work 'atack' under command
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack 22375 apache 15 0 964 560 472 S 0.3 0.0 0:04.21 atack 22858 apache 15 0 964 560 472 S 0.3 0.0 0:02.87 atack 22997 apache 15 0 964 560 472 S 0.3 0.0 0:04.11 atack 22999 apache 15 0 964 560 472 S 0.3 0.0 0:02.22 atack 23007 apache 15 0 964 560 472 S 0.3 0.0 0:03.79 atack 23099 apache 15 0 964 556 472 S 0.3 0.0 0:02.18 atack 23101 apache 15 0 964 556 472 S 0.3 0.0 0:02.48 atack 23108 apache 15 0 964 556 472 S 0.3 0.0 0:03.59 atack 23109 apache 15 0 964 556 472 S 0.3 0.0 0:02.75 atack 23112 apache 15 0 972 504 412 S 0.3 0.0 0:04.70 atack 23115 apache 15 0 964 556 472 S 0.3 0.0 0:03.75 atack 23116 apache 15 0 964 556 472 S 0.3 0.0 0:02.80 atack 23121 apache 15 0 972 504 412 S 0.3 0.0 0:03.79 atack 23384 apache 15 0 964 556 472 S 0.3 0.0 0:01.63 atack 23389 apache 15 0 964 556 472 S 0.3 0.0 0:03.52 atack 23392 apache 15 0 964 556 472 S 0.3 0.0 0:01.61 atack 23397 apache 15 0 964 556 472 S 0.3 0.0 0:01.62 atack 23405 apache 15 0 964 556 472 S 0.3 0.0 0:03.64 atack
When i 'ps -ef' i can see many lines as below;
apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100 apache 24292 23378 0 11:00 ? 00:00:01 ./atack 100 apache 24335 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24344 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24347 23378 0 11:02 ? 00:00:00 ./atack 100 apache 24358 23378 0 11:04 ? 00:00:00 ./atack 100
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
it's possible your box is attacked, has been compromised.. of it's possible that it's also being slammed by some sort of potential attack/hack. regarding the apache app, what do the log files say... what apps do you have running on the apche server? are these apps home grown, or installed from some public source?
do the research online to see what kind of attack you might have...
it might be that your box is completely safe...
you might also track/monitor any kind of attempt at the box communicating with other ip addresses that you aren't using....
doing a complete reinstall is a draconian measure and may not be called for...
your mileage might vary...
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org]On Behalf Of Linux Advocate Sent: Tuesday, June 02, 2009 8:23 PM To: CentOS mailing list Subject: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
Guys, apache cpus usage is hitting 100% sometimes ( to such an extent that its very noticeable) on a box with just 8 users or so.
i m getting this when i run 'top'. The worrying thing is seeing the work 'atack' under command
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack 22375 apache 15 0 964 560 472 S 0.3 0.0 0:04.21 atack 22858 apache 15 0 964 560 472 S 0.3 0.0 0:02.87 atack 22997 apache 15 0 964 560 472 S 0.3 0.0 0:04.11 atack 22999 apache 15 0 964 560 472 S 0.3 0.0 0:02.22 atack 23007 apache 15 0 964 560 472 S 0.3 0.0 0:03.79 atack 23099 apache 15 0 964 556 472 S 0.3 0.0 0:02.18 atack 23101 apache 15 0 964 556 472 S 0.3 0.0 0:02.48 atack 23108 apache 15 0 964 556 472 S 0.3 0.0 0:03.59 atack 23109 apache 15 0 964 556 472 S 0.3 0.0 0:02.75 atack 23112 apache 15 0 972 504 412 S 0.3 0.0 0:04.70 atack 23115 apache 15 0 964 556 472 S 0.3 0.0 0:03.75 atack 23116 apache 15 0 964 556 472 S 0.3 0.0 0:02.80 atack 23121 apache 15 0 972 504 412 S 0.3 0.0 0:03.79 atack 23384 apache 15 0 964 556 472 S 0.3 0.0 0:01.63 atack 23389 apache 15 0 964 556 472 S 0.3 0.0 0:03.52 atack 23392 apache 15 0 964 556 472 S 0.3 0.0 0:01.61 atack 23397 apache 15 0 964 556 472 S 0.3 0.0 0:01.62 atack 23405 apache 15 0 964 556 472 S 0.3 0.0 0:03.64 atack
When i 'ps -ef' i can see many lines as below;
apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100 apache 24292 23378 0 11:00 ? 00:00:01 ./atack 100 apache 24335 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24344 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24347 23378 0 11:02 ? 00:00:00 ./atack 100 apache 24358 23378 0 11:04 ? 00:00:00 ./atack 100
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Tue, Jun 02, 2009 at 09:34:55PM -0700, bruce wrote:
it's possible your box is attacked, has been compromised.. of it's possible that it's also being slammed by some sort of potential attack/hack. regarding the apache app, what do the log files say... what apps do you have running on the apche server? are these apps home grown, or installed from some public source?
He has multiple occurances of a process named "atack", each running with an argument of 100. Looks like a DoS to me.
do the research online to see what kind of attack you might have...
It's irrelevant except as a learning exercise in forensics.
it might be that your box is completely safe...
You're kidding, right?
you might also track/monitor any kind of attempt at the box communicating with other ip addresses that you aren't using....
The longer that box stays on the net the more potential damage it can (and most likely *will* do).
doing a complete reinstall is a draconian measure and may not be called for...
You're kidding, right?
John
nope...
not kidding... the majority of windows based attacks on an apache system running on linux systems are obnoxiousm but not harmful... the kinds of attacks that are looking to exploit windows buffer overflows are harmless to linux systems..
this isn't to say that all windows attacks are harmless, but this has been my experience, as well as what i've seen in the lit.
if you have other information regarding windows attaks on webservers, that also impact linux boxes, please share the relevant websites, describing the attack vectors.. i'd be interested in checking out the articles as would others...
but go ahead and reply to me online, as others might be interested in this thread as well...
-----Original Message----- From: John R. Dennison [mailto:jrd@gerdesas.com] Sent: Tuesday, June 02, 2009 9:41 PM To: bruce Cc: 'CentOS mailing list' Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
On Tue, Jun 02, 2009 at 09:34:55PM -0700, bruce wrote:
it's possible your box is attacked, has been compromised.. of it's
possible
that it's also being slammed by some sort of potential attack/hack. regarding the apache app, what do the log files say... what apps do you
have
running on the apche server? are these apps home grown, or installed from some public source?
He has multiple occurances of a process named "atack", each running with an argument of 100. Looks like a DoS to me.
do the research online to see what kind of attack you might have...
It's irrelevant except as a learning exercise in forensics.
it might be that your box is completely safe...
You're kidding, right?
you might also track/monitor any kind of attempt at the box communicating with other ip addresses that you aren't using....
The longer that box stays on the net the more potential damage it can (and most likely *will* do).
doing a complete reinstall is a draconian measure and may not be called for...
You're kidding, right?
John
-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when asked to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently not fun.
Bruce:
I think you are misunderstanding something. He showed a process listing of processes running on his server. Those were not apache processes being attacked from the outside. They were rogue processes running on his machine.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of bruce Sent: Tuesday, June 02, 2009 11:49 PM Cc: 'CentOS mailing list' Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
nope...
not kidding... the majority of windows based attacks on an apache system running on linux systems are obnoxiousm but not harmful... the kinds of attacks that are looking to exploit windows buffer overflows are harmless to linux systems..
this isn't to say that all windows attacks are harmless, but this has been my experience, as well as what i've seen in the lit.
if you have other information regarding windows attaks on webservers, that also impact linux boxes, please share the relevant websites, describing the attack vectors.. i'd be interested in checking out the articles as would others...
but go ahead and reply to me online, as others might be interested in this thread as well...
-----Original Message----- From: John R. Dennison [mailto:jrd@gerdesas.com] Sent: Tuesday, June 02, 2009 9:41 PM To: bruce Cc: 'CentOS mailing list' Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
On Tue, Jun 02, 2009 at 09:34:55PM -0700, bruce wrote:
it's possible your box is attacked, has been compromised.. of it's
possible
that it's also being slammed by some sort of potential attack/hack. regarding the apache app, what do the log files say... what
apps do you have
running on the apche server? are these apps home grown, or
installed from
some public source?
He has multiple occurances of a process named "atack", each running with an argument of 100. Looks like a DoS to me.
do the research online to see what kind of attack you might have...
It's irrelevant except as a learning exercise in forensics.
it might be that your box is completely safe...
You're kidding, right?
you might also track/monitor any kind of attempt at the box
communicating
with other ip addresses that you aren't using....
The longer that box stays on the net the more potential damage it can (and most likely *will* do).
doing a complete reinstall is a draconian measure and may
not be called
for...
You're kidding, right?
John-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when asked to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently not fun.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
neil...
the ps he showed, showed the 'atack' processes being run by the apache user...
i'm incined to agree that he should take the machine offline, but i don't know what the 'atack' processes are, and unless his system is really f*ed up.. i'm inclined to think the processs is something on his server...
now, how it got there is a curious issue that he's going to have to address..
but this is why i specifically asked the kinds of web apps he's running on his server...
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org]On Behalf Of Neil Aggarwal Sent: Tuesday, June 02, 2009 10:03 PM To: 'CentOS mailing list' Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
Bruce:
I think you are misunderstanding something. He showed a process listing of processes running on his server. Those were not apache processes being attacked from the outside. They were rogue processes running on his machine.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of bruce Sent: Tuesday, June 02, 2009 11:49 PM Cc: 'CentOS mailing list' Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
nope...
not kidding... the majority of windows based attacks on an apache system running on linux systems are obnoxiousm but not harmful... the kinds of attacks that are looking to exploit windows buffer overflows are harmless to linux systems..
this isn't to say that all windows attacks are harmless, but this has been my experience, as well as what i've seen in the lit.
if you have other information regarding windows attaks on webservers, that also impact linux boxes, please share the relevant websites, describing the attack vectors.. i'd be interested in checking out the articles as would others...
but go ahead and reply to me online, as others might be interested in this thread as well...
-----Original Message----- From: John R. Dennison [mailto:jrd@gerdesas.com] Sent: Tuesday, June 02, 2009 9:41 PM To: bruce Cc: 'CentOS mailing list' Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
On Tue, Jun 02, 2009 at 09:34:55PM -0700, bruce wrote:
it's possible your box is attacked, has been compromised.. of it's
possible
that it's also being slammed by some sort of potential attack/hack. regarding the apache app, what do the log files say... what
apps do you have
running on the apche server? are these apps home grown, or
installed from
some public source?
He has multiple occurances of a process named "atack", each running with an argument of 100. Looks like a DoS to me.
do the research online to see what kind of attack you might have...
It's irrelevant except as a learning exercise in forensics.
it might be that your box is completely safe...
You're kidding, right?
you might also track/monitor any kind of attempt at the box
communicating
with other ip addresses that you aren't using....
The longer that box stays on the net the more potential damage it can (and most likely *will* do).
doing a complete reinstall is a draconian measure and may
not be called
for...
You're kidding, right?
John-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when asked to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently not fun.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Bruce:
i'm inclined to think the processs is something on his server...
now, how it got there is a curious issue that he's going to have to address..
This is precisely the point. An unauthorized user currently has the ability to run processed on the machine. We do not know what they have already done or will do to the machine. We have to assume the entire machine is suspect and therefore it needs to be wiped.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
neil...
you state that "..An unauthorized user currently has the ability to run processed on the machine...."
how do we know that.. did i miss something in an earlier thread.. don't get me wrong, you might know more on this thread than the few msgs i saw... al i saw was that there was the 'atack' process being run...
do we know how it got there?
did he say he didn't know what the hell the process was and that he didn't put it there? also, did he ever say if he was the only one to put things on the box.. (ie, a friend of his didn't put it there.. )
as an aside? did he say if he even looked on the net for anything related to this??
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org]On Behalf Of Neil Aggarwal Sent: Tuesday, June 02, 2009 10:21 PM To: 'CentOS mailing list' Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
Bruce:
i'm inclined to think the processs is something on his server...
now, how it got there is a curious issue that he's going to have to address..
This is precisely the point. An unauthorized user currently has the ability to run processed on the machine. We do not know what they have already done or will do to the machine. We have to assume the entire machine is suspect and therefore it needs to be wiped.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Bruce:
you state that "..An unauthorized user currently has the ability to run processed on the machine...."
how do we know that..
The original poster stated he did know how what the process was. He stated he believed the machine was being attacked. He asked for advice from the community on how to handle the situation.
The original poster's statments imply it was not put there by an authorized user. Someone does not just casually assume a machine has been hacked. They have a reason for suspecting it.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
My replies below.... i m just so down in the dumps now....aaahhhhh
----- Original Message ----
From: Neil Aggarwal neil@JAMMConsulting.com To: CentOS mailing list centos@centos.org Sent: Wednesday, June 3, 2009 1:38:05 PM Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
The original poster stated he did know how what the process was. He stated he believed the machine was being attacked. He asked for advice from the community on how to handle the situation.
yes. this was and is still my understanding. This was what 'top' showed...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack 22375 apache 15 0 964 560 472 S 0.3 0.0 0:04.21 atack 22858 apache 15 0 964 560 472 S 0.3 0.0 0:02.87 atack
'ps -ef' showed
apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100 apache 24292 23378 0 11:00 ? 00:00:01 ./atack 100 apache 24335 23378 0 11:01 ? 00:00:00 ./atack 100
The original poster's statments imply it was not put there by an authorized user.
yes , no one but me has access to the machine.
Someone does not just casually assume a machine has been hacked. They have a reason for suspecting it.
Applications running;
1 - horde groupware webmail edition, just the framework though. 2 - phpmyadmin 3 - postfixadmin 4 - postfix 5 - dovecot 6. fail2ban 7. monit
2 -> 7 i installed from the repos.
The centos box was running 5.2 when i first noticed the 'slowness'. i then updated to 5.3 hoping that the problem would go away.
i am not worried abt reinstalling ( i loathe doing it ) but my worry here ( as some of you have accurately pointed out ) is that the 'issue' will repeat again bcos i just downt know what happened. I m just surprised that a centos box was compromised.
The box is unplugged now.
Any more ideas?
Regards, Maco.
Maco:
i am not worried abt reinstalling ( i loathe doing it ) but my worry here ( as some of you have accurately pointed out ) is that the 'issue' will repeat again bcos i just downt know what happened. I m just surprised that a centos box was compromised.
If you are only running software installed from the repos, the best thing to do is to wipe and reinstall the machine from scratch. Make sure it has the latest versions of everything you are using.
Like I said earlier, it is going to be very hard to determine exactly how it was hacked. Hopefully, whatever the hacker used has been patched.
If it is a new exploit, any CentOS server is vulnerable, not just yours. I assume the hacker would compromise more machines than just yours. The hole will eventually be discovered and fixed.
As I said before, nothing is 100% secure.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
On Wed, Jun 3, 2009 at 9:22 AM, Linux Advocate linuxhousedn@yahoo.com wrote:
i am not worried abt reinstalling ( i loathe doing it ) but my worry here ( as some of you have accurately pointed out ) is that the 'issue' will repeat again bcos i just downt know what happened. I m just surprised that a centos box was compromised.
The box is unplugged now.
Any more ideas?
Keep the old OS data for forensic analysis, but build a fresh install with only the essential services needed to host the web site, not manage it.
It may be a lot of work, but going forward think about using Xen PV domains for the edge web hosts on vlans in a dmz.
You can mount the web data via a read-only NFS share through the DMZ firewall, and have 2 hosts balanced and a 3rd as a "hot-spare" host in case any of the first two get compromised. Even better build a web host image, take LVM snapshots of it and have Xen boot those!
Software inheritenly has bugs, and some of those bugs will lead to security compromises. Keep your software up to date, only install necessary services, build your security in layers and have a backup plan.
-Ross
On Wed, 2009-06-03 at 06:29 -0700, Linux Advocate wrote:
<snip>
i tried googling for 'centos apache atack" but did not get anything substantial. i tried locating a binary file called ' atack' but got nothing.
Just an FYI to all those who may not know:
$ cat test.c #include <stdlib.h> #include <stdio.h> #include <string.h> main(int argc, char *argv[]) { sleep(15); strcpy(argv[0],"test.c"); sleep(15); exit(0); }
$ cc test.c [wild-bill@centos501 ~]$ ./a.out& [2] 7359 [wild-bill@centos501 ~]$ ps -ef|tail -4 500 7323 4104 0 10:52 ? 00:00:00 spamd child 500 7359 4025 0 10:54 pts/0 00:00:00 ./a.out 500 7360 4025 0 10:54 pts/0 00:00:00 ps -ef 500 7361 4025 0 10:54 pts/0 00:00:00 tail -4 [wild-bill@centos501 ~]$ sleep 15;ps -ef|tail -4 500 7323 4104 0 10:52 ? 00:00:00 spamd child 500 7359 4025 0 10:54 pts/0 00:00:00 test.c 500 7363 4025 0 10:54 pts/0 00:00:00 ps -ef 500 7364 4025 0 10:54 pts/0 00:00:00 tail -4
I haven't checked in a long time, but maybe there's some stuff in process group headers that might give a clue to follow? Been a *long* time since I dinked with that stuff, so I'm not sure.
One thing to check for is anything with an suid bit set that is owner apache (again a long time, but I think that will do it) that you suspect is "wrong". Sometime clues reside in timestamps on the executables. Might need to do your snooping in single-user mode off a recovery CD since well-crafted attacks hide themselves and overlay commands that might be used to detect them.
Barring all else, an rpm -qa --last will show installs by date and a --verify might yield some clues. You can "find" with various time checks (-newer or -mtime?) to see all files and directories that have been changed since the last rpm activity prior to the detection of the problem. However, these can also be modifed to reduce the chance of detection.
<snip>
HTH
Bill:
Just an FYI to all those who may not know:
$ cat test.c #include <stdlib.h> #include <stdio.h> #include <string.h> main(int argc, char *argv[]) { sleep(15); strcpy(argv[0],"test.c"); sleep(15); exit(0); }
That is a very cool demonstration. Thanks for the info.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
On Wed, 2009-06-03 at 11:06 -0400, William L. Maltby wrote:
<snip>
I just thought of this too.
There are two IDs tracked by the system. Effective (EUID) and the real ID (UID). If the process has changed UID, by either suid bit or by program call (I think it has to start as root for that to happen?), you can run ps with a flag that will show you the real and/or EUID.
That might provide a clue as well.
HTH
On Tue, Jun 02, 2009 at 09:48:41PM -0700, bruce wrote:
not kidding... the majority of windows based attacks on an apache system running on linux systems are obnoxiousm but not harmful... the kinds of attacks that are looking to exploit windows buffer overflows are harmless to linux systems..
this isn't to say that all windows attacks are harmless, but this has been my experience, as well as what i've seen in the lit.
if you have other information regarding windows attaks on webservers, that also impact linux boxes, please share the relevant websites, describing the attack vectors.. i'd be interested in checking out the articles as would others...
Not to be rude but what you are rambling on about?
He's running an apache instance on cent5. He has processes he can not readily identify running under apache named "atack"; where does "windows" come into the equation? What the processes are specifically doing is secondary to the problem at hand, which is that the processes exist in the first place.
Please, enlighten me as to how you can think that his box has not been compromised. Please, enlighten me as to how he (or you) can gauge the extent of the compromise (assuming no HIDS in use on the server).
I stand by my previous advice - the box is compromised, can not be trusted, and as a responsible admin he should be working on re-installing it, evaluating what web-apps he had running that led to this in the first place and taking the appropriate steps to ensure it does not happen again.
John
you and i agreee on him figuring out what web apps are causing the issues.. or in fact, exactly what the 'atack' process is? i didn't see the initial threads.. was this simething that he discussed? did he say what the arack process was doing?
my only point, was that reinstalling wotjout understanding what was/is going on is a draconian step.. does it resolve the issue.. sire.. does it get to what might have been the cause.. not in my opinion...
but hey.. there are different ways of approaching a problem...
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org]On Behalf Of John R. Dennison Sent: Tuesday, June 02, 2009 10:10 PM To: CentOS mailing list Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
On Tue, Jun 02, 2009 at 09:48:41PM -0700, bruce wrote:
not kidding... the majority of windows based attacks on an apache system running on linux systems are obnoxiousm but not harmful... the kinds of attacks that are looking to exploit windows buffer overflows are harmless
to
linux systems..
this isn't to say that all windows attacks are harmless, but this has been my experience, as well as what i've seen in the lit.
if you have other information regarding windows attaks on webservers, that also impact linux boxes, please share the relevant websites, describing
the
attack vectors.. i'd be interested in checking out the articles as would others...
Not to be rude but what you are rambling on about?
He's running an apache instance on cent5. He has processes he can not readily identify running under apache named "atack"; where does "windows" come into the equation? What the processes are specifically doing is secondary to the problem at hand, which is that the processes exist in the first place.
Please, enlighten me as to how you can think that his box has not been compromised. Please, enlighten me as to how he (or you) can gauge the extent of the compromise (assuming no HIDS in use on the server).
I stand by my previous advice - the box is compromised, can not be trusted, and as a responsible admin he should be working on re-installing it, evaluating what web-apps he had running that led to this in the first place and taking the appropriate steps to ensure it does not happen again.
John
-- "I'm sorry but our engineers do not have phones." As stated by a Network Solutions Customer Service representative when asked to be put through to an engineer.
"My other computer is your windows box." Ralf Hildebrandt <sxem> trying to play sturgeon while it's under attack is apparently not fun.
Bruce:
my only point, was that reinstalling wotjout understanding what was/is going on is a draconian step.. does it resolve the issue.. sire.. does it get to what might have been the cause.. not in my opinion...
This point seems valid.
If you do not understand why and how the machine was compromised, there is no way to be sure a reinstall will plug the security hole.
The reality of the matter is that it is extremely unlikely that he could figure out precisely how the machine was compromised. There is just not going to be a smoking gun that says the hacker did A, B, and C and got in.
It would be prudent to review his web code to see if he did something in an insecure way. If his code is open to attack, it will be so even if he puts it on a new machine.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
On Wed, Jun 03, 2009 at 12:30:10AM -0500, Neil Aggarwal wrote:
It would be prudent to review his web code to see if he did something in an insecure way. If his code is open to attack, it will be so even if he puts it on a new machine.
Hence my statements to evaluate the web-apps he has running :)
I will bet dollars to donuts he had a web app with a known issue that was not patched. Also goes back to my previous statement of fully patching.
John
On Wed, 2009-06-03 at 00:46 -0500, John R. Dennison wrote:
On Wed, Jun 03, 2009 at 12:30:10AM -0500, Neil Aggarwal wrote:
It would be prudent to review his web code to see if he did something in an insecure way. If his code is open to attack, it will be so even if he puts it on a new machine.
Hence my statements to evaluate the web-apps he has running :)
I will bet dollars to donuts he had a web app with a known issue that was not patched. Also goes back to my previous statement of fully patching.
--- Dollars to Donuts ehhh??? How many donuts you think it will take to pay for legal costs and clean up if there are customer data on the machine? I think right about now I would: 1. Notify Risk Management and Your Compliancy Officer. 2. Take it off the network connections. 3. Do a live rsync and dd image + ram copy = running processes/hidden. 4. Same as 3. but with the machine off. 5. The company attorney needs to be notified. 6. By State and Federal Law in the US you have so many days to report incidents like this to users (customers) and law enforcement.
JohnStanley
On Wed, 2009-06-03 at 01:57 -0400, JohnS wrote:
On Wed, 2009-06-03 at 00:46 -0500, John R. Dennison wrote:
On Wed, Jun 03, 2009 at 12:30:10AM -0500, Neil Aggarwal wrote:
It would be prudent to review his web code to see if he did something in an insecure way. If his code is open to attack, it will be so even if he puts it on a new machine.
Hence my statements to evaluate the web-apps he has running :)
I will bet dollars to donuts he had a web app with a known issue that was not patched. Also goes back to my previous statement of fully patching.
Dollars to Donuts ehhh??? How many donuts you think it will take to pay for legal costs and clean up if there are customer data on the machine? I think right about now I would:
- Notify Risk Management and Your Compliancy Officer.
- Take it off the network connections.
- Do a live rsync and dd image + ram copy = running processes/hidden.
- Same as 3. but with the machine off.
- The company attorney needs to be notified.
- By State and Federal Law in the US you have so many days to report
incidents like this to users (customers) and law enforcement.
If, by step 4, you mean remove the drive[1], stick it into USB enclosure, make a copy of it, then stick the original into a plastic bag in full view of a witness[2] then give it to them, I agree wholeheartedly[3]. I've been through this before and this is, IMHO[4] a safer way to operate.
-I
[1] Assuming no RAID. If you have RAID, you can go to a separate box and make a live backup via: goodhost# ssh badhost '(cat /dev/sda)' > badhost-sda.ddout [2] Your manager or corporate counsel will do in this example. Better if its both. [3] This does *NOT* constitute legal advice. Talk to your corporate counsel before taking action, as this may constitute a criminal matter. [4] See [3] above.
On Wed, Jun 03, 2009 at 01:57:20AM -0400, JohnS wrote:
Dollars to Donuts ehhh??? How many donuts you think it will take to pay for legal costs and clean up if there are customer data on the machine? I think right about now I
4 chocolate eclairs should cover it :)
But seriously...
would:
- Notify Risk Management and Your Compliancy Officer.
- Take it off the network connections.
- Do a live rsync and dd image + ram copy = running processes/hidden.
- Same as 3. but with the machine off.
- The company attorney needs to be notified.
- By State and Federal Law in the US you have so many days to report
incidents like this to users (customers) and law enforcement.
While the specifics vary from company to company depending on your corporate escalation procedures the above points are very valid and would of course need to be properly followed as required by your corporate entity.
My comment regarding donuts was intended to be flippant and add a light side to the conversation; I assumed from the start that the original poster would follow his corporations established policy on notification and escalation as required.
John
On Wed, 2009-06-03 at 02:04 -0500, John R. Dennison wrote:
On Wed, Jun 03, 2009 at 01:57:20AM -0400, JohnS wrote:
Dollars to Donuts ehhh??? How many donuts you think it will take to pay for legal costs and clean up if there are customer data on the machine? I think right about now I
4 chocolate eclairs should cover it :)
But seriously...
would:
- Notify Risk Management and Your Compliancy Officer.
- Take it off the network connections.
- Do a live rsync and dd image + ram copy = running processes/hidden.
- Same as 3. but with the machine off.
- The company attorney needs to be notified.
- By State and Federal Law in the US you have so many days to report
incidents like this to users (customers) and law enforcement.
While the specifics vary from company to company depending on your corporate escalation procedures the above points are very valid and would of course need to be properly followed as required by your corporate entity.
My comment regarding donuts was intended to be flippant and add a light side to the conversation; I assumed from the start that the original poster would follow his corporations established policy on notification and escalation as required.
--- I guess all we can do is hope. No offense taken here though.
JohnStanley
It would be prudent to review his web code to see if he did something in an insecure way. If his code is open to attack, it will be so even if he puts it on a new machine.
Hence my statements to evaluate the web-apps he has running :)
I will bet dollars to donuts he had a web app with a known issue that was not patched. Also goes back to my previous statement of fully patching.
Dollars to Donuts ehhh??? How many donuts you think it will take to pay for legal costs and clean up if there are customer data on the machine? I think right about now I would:
- Notify Risk Management and Your Compliancy Officer.
- Take it off the network connections.
- Do a live rsync and dd image + ram copy = running processes/hidden.
- Same as 3. but with the machine off.
- The company attorney needs to be notified.
- By State and Federal Law in the US you have so many days
to report incidents like this to users (customers) and law enforcement.
I would say, if he is local to the datacenter, pull the machine. Take it home and analyze what is going on with it. Reinstalling does nothing to keep it from happening as soon as it is back on the net.
The admin must find out what it is. I think we all agree on somethings..
1- disconnect from the internet 2- back up all data 3- virus/trojan scan all data backed up 4 - after figuring out what is happening and how it has happened.... 4a - root kit? Other security programs? Virus/trojan check again. 4c- check all logs of any kind for any sort of key on anything sent out from the server. 5- reinstall, patch, readd data 6- check for issues regarding the original issue.
I think everyone is on the same page but does not know it. I think every single person reading this would love to see not only the resolution but what caused it and any info on preventing it.
on 6-3-2009 1:15 PM Bob Hoffman spake the following:
It would be prudent to review his web code to see if he did something in an insecure way. If his code is open to attack, it will be so even if he puts it on a new machine.
Hence my statements to evaluate the web-apps he has running :)
I will bet dollars to donuts he had a web app with a known issue that was not patched. Also goes back to my previous statement of fully patching.
Dollars to Donuts ehhh??? How many donuts you think it will take to pay for legal costs and clean up if there are customer data on the machine? I think right about now I would:
- Notify Risk Management and Your Compliancy Officer.
- Take it off the network connections.
- Do a live rsync and dd image + ram copy = running processes/hidden.
- Same as 3. but with the machine off.
- The company attorney needs to be notified.
- By State and Federal Law in the US you have so many days
to report incidents like this to users (customers) and law enforcement.
I would say, if he is local to the datacenter, pull the machine. Take it home and analyze what is going on with it. Reinstalling does nothing to keep it from happening as soon as it is back on the net.
The admin must find out what it is. I think we all agree on somethings..
1- disconnect from the internet 2- back up all data 3- virus/trojan scan all data backed up 4 - after figuring out what is happening and how it has happened.... 4a - root kit? Other security programs? Virus/trojan check again. 4c- check all logs of any kind for any sort of key on anything sent out from the server. 5- reinstall, patch, readd data 6- check for issues regarding the original issue.
I think everyone is on the same page but does not know it. I think every single person reading this would love to see not only the resolution but what caused it and any info on preventing it.
Looking at some of the apps he was running, several of them have had vulnerabilities in the past like phpmysqladmin. I see script kiddie runs at that almost every day, along with runs at horde, roundcube webmail, and sql injection and buffer overflow attempts against apache.
Everything on the internet is a target.
on 6-2-2009 10:18 PM bruce spake the following:
you and i agreee on him figuring out what web apps are causing the issues.. or in fact, exactly what the 'atack' process is? i didn't see the initial threads.. was this simething that he discussed? did he say what the atack process was doing?
Who cares what it was doing? He stated he didn't know what it was. It could be sending spam or making tea, it doesn't matter. It is running without his knowledge.
my only point, was that reinstalling without understanding what was/is going on is a draconian step.. does it resolve the issue.. sire.. does it get to what might have been the cause.. not in my opinion...
Attack forensics is an art. There are people that make large sums of money doing this because it is difficult. Does he have the time/resources to see what happened, or does he just need to get his site up and working in the least amount of time?
but hey.. there are different ways of approaching a problem...
Either way you want to look at it, the box needs to at a minimum get off the net. If the system only has remote access, it needs to be booted from some sort of rescue system to isolate the base from the running system. If he has local access, then all the work can be done from a local console. Back up anything you want, but don't just restore everything to the rebuilt system, but check everything. Then you can analyze, backup, wipe, pray, piss and moan, drink, or whatever strikes your fancy. Just get the system off the internet until it is not a (possible) threat anymore.
and if you don't figure out what caused the issue...
there's not a dammed reason to think you wouldn't do the same thing and get in the same dam situation when you reinstall...
i'm not quibbling with removing the box from the net... i've simply stated that just going straight to reinstall doesn't resolve the potential reoccurance of the issue..
in his case though, it now appears that he's got a great deal more information regarding the hack, and that he can proceed to figure out what happened.. or he might just reinstall!
peace
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org]On Behalf Of Scott Silva Sent: Wednesday, June 03, 2009 10:57 AM To: centos@centos.org Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
on 6-2-2009 10:18 PM bruce spake the following:
you and i agreee on him figuring out what web apps are causing the issues.. or in fact, exactly what the 'atack' process is? i didn't see the initial threads.. was this simething that he discussed? did he say what the atack process was doing?
Who cares what it was doing? He stated he didn't know what it was. It could be sending spam or making tea, it doesn't matter. It is running without his knowledge.
my only point, was that reinstalling without understanding what was/is going on is a draconian step.. does it resolve the issue.. sire.. does it get to what might have been the cause.. not in my opinion...
Attack forensics is an art. There are people that make large sums of money doing this because it is difficult. Does he have the time/resources to see what happened, or does he just need to get his site up and working in the least amount of time?
but hey.. there are different ways of approaching a problem...
Either way you want to look at it, the box needs to at a minimum get off the net. If the system only has remote access, it needs to be booted from some sort of rescue system to isolate the base from the running system. If he has local access, then all the work can be done from a local console. Back up anything you want, but don't just restore everything to the rebuilt system, but check everything. Then you can analyze, backup, wipe, pray, piss and moan, drink, or whatever strikes your fancy. Just get the system off the internet until it is not a (possible) threat anymore.
----- Original Message ----
From: bruce bedouglas@earthlink.net To: CentOS mailing list centos@centos.org Sent: Thursday, June 4, 2009 3:20:24 AM Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
and if you don't figure out what caused the issue...
working on it bro :)
one of the pointers here was to look at alias directives in apache...
when i run httpd -S i get these errors...
[Sat Jun 13 15:14:09 2009] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 11 will probably never match because it overlaps an earlier Alias. [Sat Jun 13 15:14:09 2009] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 12 will probably never match because it overlaps an earlier Alias.
the contents of /etc/httpd/conf.d/phpmyadmin.conf are;
# Web application to manage MySQL #
<Directory "/usr/share/phpmyadmin"> Order Deny,Allow Deny from all Allow from 127.0.0.1 </Directory>
Alias /phpmyadmin /usr/share/phpmyadmin <--- 1 Alias /phpMyAdmin /usr/share/phpmyadmin < --- 2 is this normal ??? Alias /mysqladmin /usr/share/phpmyadmin
Is it normal to have these lines?
there's not a dammed reason to think you wouldn't do the same thing and get in the same dam situation when you reinstall...
agreed.
when i run httpd -S i get these errors...
[Sat Jun 13 15:14:09 2009] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 11 will probably never match because it overlaps an earlier Alias. [Sat Jun 13 15:14:09 2009] [warn] The Alias directive in /etc/httpd/conf.d/phpmyadmin.conf at line 12 will probably never match because it overlaps an earlier Alias.
the contents of /etc/httpd/conf.d/phpmyadmin.conf are;
# Web application to manage MySQL #
<Directory "/usr/share/phpmyadmin"> Order Deny,Allow Deny from all Allow from 127.0.0.1
</Directory>
Alias /phpmyadmin /usr/share/phpmyadmin <--- 1 Alias /phpMyAdmin /usr/share/phpmyadmin < --- 2 is this normal ??? Alias /mysqladmin /usr/share/phpmyadmin
Is it normal to have these lines?
Depending on your setup, yes it can be. The "Alias" directives are there so that when you type in "http://www.mysite.com/phpmyadmin" Apache will redirect the request to "/usr/share/phpmyadmin".
What this does is allow you to keep scripts outside of a website's directory structure. I use them with PHPMyAdmin to primarily prevent tampering by my various users but it also makes it easier to update/patch the app(s) when needed.
On Wednesday 03 June 2009 06:09:37 John R. Dennison wrote:
He's running an apache instance on cent5. He has processes he can not readily identify running under apache named "atack"; where does "windows" come into the equation?
Several of the links returned by google have the following info:
IIS WebDAV Exploit, I think one of the agobot worms tries to use it to get into Windows boxes.
Anne
Anne Wilson wrote:
On Wednesday 03 June 2009 06:09:37 John R. Dennison wrote:
He's running an apache instance on cent5. He has processes he can not readily identify running under apache named "atack"; where does "windows" come into the equation?
Several of the links returned by google have the following info:
IIS WebDAV Exploit, I think one of the agobot worms tries to use it to get into Windows boxes.
AGAIN: He has processes running as the apache user on his Linux box which he cannot identify.
What makes you think that this is an attack on a WINDOWS system?
Ralph
On Wednesday 03 June 2009 12:44:58 Ralph Angenendt wrote:
where does "windows" come into the equation?
The question I replied to was "where does "windows" come into the equation?".
Anne
Anne Wilson wrote:
On Wednesday 03 June 2009 12:44:58 Ralph Angenendt wrote:
where does "windows" come into the equation?
No, I did not write that.
The question I replied to was "where does "windows" come into the equation?".
And I asked what made you think that this had anything to do with windows.
Ralph
On Wednesday 03 June 2009 14:09:35 Ralph Angenendt wrote:
Anne Wilson wrote:
On Wednesday 03 June 2009 12:44:58 Ralph Angenendt wrote:
where does "windows" come into the equation?
No, I did not write that.
True. An error in snipping, somewhere.
The question I replied to was "where does "windows" come into the
equation?".
And I asked what made you think that this had anything to do with windows.
And I never said it had, other than the quote which says it is aimed at windows servers. I pass no opinion.
Anne
Anne Wilson wrote:
On Wednesday 03 June 2009 06:09:37 John R. Dennison wrote:
He's running an apache instance on cent5. He has processes he can not readily identify running under apache named "atack"; where does "windows" come into the equation?Several of the links returned by google have the following info:
IIS WebDAV Exploit, I think one of the agobot worms tries to use it to get into Windows boxes.
well when I google I have to conclude that this all has to do with helicopters.
----- Original Message ----
From: Anne Wilson cannewilson@googlemail.com
On Wednesday 03 June 2009 06:09:37 John R. Dennison wrote:
He's running an apache instance on cent5. He has processes he can not readily identify running under apache named "atack"; where does "windows" come into the equation?Several of the links returned by google have the following info:
IIS WebDAV Exploit, I think one of the agobot worms tries to use it to get into Windows boxes.
Anne
Anne, i m running apache on a centos box. is centos still susceptible?
On Wednesday 03 June 2009 14:24:43 Linux Advocate wrote:
----- Original Message ----
From: Anne Wilson cannewilson@googlemail.com
On Wednesday 03 June 2009 06:09:37 John R. Dennison wrote:
He's running an apache instance on cent5. He has processes he can not readily identify running under apache named "atack"; where does "windows" come into the equation?Several of the links returned by google have the following info:
IIS WebDAV Exploit, I think one of the agobot worms tries to use it to get into Windows boxes.
Anne
Anne, i m running apache on a centos box. is centos still susceptible?
No idea, I'm afraid. I know b* all about this. I was merely trying to avoid a side-issue in the discussion by pointing out how windows got mentioned. Sorry.
Anne
----- Original Message ----
From: John R. Dennison jrd@gerdesas.com
I stand by my previous advice - the box is compromised, can not be trusted, and as a responsible admin he should be working on re-installing it, evaluating what web-apps he had running that led to this in the first place and taking the appropriate steps to ensure it does not happen again.
what steps should i take. i was running centos 5.2 fully updated. the web apps or daemons i have running are from the repos. i have other mandriva boxes and they all are ok. i m just so surprised that a centos box got compromised.
Maco:
i have other mandriva boxes and they all are ok. i m just so surprised that a centos box got compromised.
If you are not doing anything silly in your server configuration, this is not a CentOS issue.
Anything *can* be hacked. It just so happens that it was your CentOS box this time.
Neil
-- Neil Aggarwal, (832)245-7314, www.JAMMConsulting.com Eliminate junk email and reclaim your inbox. Visit http://www.spammilter.com for details.
Neil Aggarwal wrote:
Maco:
i have other mandriva boxes and they all are ok. i m just so surprised that a centos box got compromised.
If you are not doing anything silly in your server configuration, this is not a CentOS issue.
Anything *can* be hacked. It just so happens that it was your CentOS box this time.
My two cents here
I'm probably stating the obvious here for many users, but ... -
For web apps installed from CentOS / EPEL /etc - modify the configuration file to change apache alias directive.
Look at your web logs sometime, whether or not you use apps like phpmyadmin or squirrelmail, you will see requests to where CentOS (and other distros) make those apps available by default.
These requests are usually either brute force attacks against those apps or trying known (often patched if you keep yum up to date) exploits for them.
By changing the alias in the configuration file, when a new exploit is found and the script kiddies launch their scripts against the web, they'll likely miss your box unless they know where to send the request to.
Yes, that's security by obscurity, but security by obscurity will protect you from most script kitty attacks, and may prevent you from being owned by a close to zero day exploit.
For things like squirrelmail, don't allow it over http, require it be done over https to avoid any sniffing (open networks at coffee shops or student labs or common places for sniffing).
I recommend using suhosin for php - and use some of the suhosin directives that lock down the php install, such as not allowing shell execution from within php.
That will break some apps (IE squirrelmail requires exec to send a message) but you can specifically enable it for certain web apps and you may be able to patch some apps to no longer need shell execution (IE I believe that squirrelmail could be patched to use php's native mail interface, maybe even easily by using phpMailer, but I've not tried).
If you look in pear and pecl, you can sometimes find native ways to do what many apps currently want shell execution for - IE if you use shell execution for ImageMagick, there's a pecl binary extension you can build to do it in pure php w/o calling exec() thus allowing you turn off exec() via suhosin.
Many web applications are (historically anyway) vulnerable to sql injection attacks. These attacks can be used to get password hashes that allow the attacker to crack user accounts and elevate their privileges within the web app. Many of web applications out there in common use have not been audited.
SQL injection can pretty much be neutered by using prepared statements, so check your web app to see if it uses prepared statements and if it doesn't, port it to use prepared statements.
I personally port them to use the pear::mdb2 abstraction layer at the same time, giving me a little more flexibility in case I ever decide I don't want to use MySQL anymore.
And for user password hashes, one thing you'll find is that there are some passwords that are very commonly used, so if all you do to make your hash is some variant of md5sum($pass . $salt) and a cracker does get the hash - he just has to look for hashes that occur often and try the passwords used frequently against those accounts.
md5sum($pass . strtolower($username) . $salt) or something like that results in unique hashes for two accounts even if the two accounts have identical passwords.
Another problem many web applications have is they want the configuration file to be writeable by the web server - and even worse, executed by the web server as a script. I do not believe that is the case for any web apps in rhel/centos or epel, but for something you grab off the web (IE the sphyder search engine) that often is the case. Any web app that has a hole can then be used to trick apache into writing to that configuration file resulting in apache then executing the malicious code.
Make damn sure those configuration files are only readable by the web server, hand edit them to make changes. If you MUST use the admin interface of those apps to change configuration, then make a database table to hold the configurations and port the app to get its configuration from the database rather than flat file that apache can write to that the web app then parses as php.
Basically, audit every app out there you plan to use - the people who write these web applications often don't take security into consideration before they upload them to their server for your consumption.
-----Original Message-----
Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
Basically, audit every app out there you plan to use - the people who write these web applications often don't take security into consideration before they upload them to their server for your consumption.
Ditto ditto ditto. And it is wise, although very time consuming, to look at all programs loaded onto your centos too. Mysql comes with a number of ways to get full access unless you go right in and change localhost/localdomain user/pass and delete the two extra accounts...
And that is just one.
Rarely, rarely, do I see a application built from security first as far as web apps. Dang scary. If you are using a popular program an exploit will be done automatically to every site that has it. Since each install uses the same pages basically, it is easy for a autobot to find them all and zero day your forums, xss your whatever, and so on.
Dang scary to leave JS on at all....even though you basically have too.
Linux Advocate wrote:
----- Original Message ----
From: John R. Dennison jrd@gerdesas.com
I stand by my previous advice - the box is compromised, can not be trusted, and as a responsible admin he should be working on re-installing it, evaluating what web-apps he had running that led to this in the first place and taking the appropriate steps to ensure it does not happen again.what steps should i take. i was running centos 5.2 fully updated. the web apps or daemons i have running are from the repos. i have other mandriva boxes and they all are ok. i m just so surprised that a centos box got compromised.
There were dozens of security updates to php and related apps since the 5.2 days. You really have to keep anything exposed to the internet up to date and using secure passwords. This almost certainly isn't a 'centos' issue. Someone probably used a default password to log into one of the php apps and exploit an old bug that let them write in a place that apache would execute something. Odds are that they didn't get root and that you'd have a chance of cleaning it if you know what you are doing, but if you have to ask for advice on a mail list you probably shouldn't try.
bruce wrote:
nope...
not kidding... the majority of windows based attacks on an apache system running on linux systems are obnoxiousm but not harmful... the kinds of attacks that are looking to exploit windows buffer overflows are harmless to linux systems..
Aha. How are active running processes on a CentOS box a "windows based attack"?
Have you looked at the first mail in this thread - those aren't logfile excerpts, those are processes running as the apache user on that box.
Ralph
htebruce wrote:
it's possible your box is attacked, has been compromised.. of it's possible that it's also being slammed by some sort of potential attack/hack. regarding the apache app, what do the log files say... what apps do you have running on the apche server? are these apps home grown, or installed from some public source?
do the research online to see what kind of attack you might have...
it might be that your box is completely safe...
you might also track/monitor any kind of attempt at the box communicating with other ip addresses that you aren't using....
doing a complete reinstall is a draconian measure and may not be called for...
your mileage might vary...
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org]On Behalf Of Linux Advocate Sent: Tuesday, June 02, 2009 8:23 PM To: CentOS mailing list Subject: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
Guys, apache cpus usage is hitting 100% sometimes ( to such an extent that its very noticeable) on a box with just 8 users or so.
i m getting this when i run 'top'. The worrying thing is seeing the work 'atack' under command
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack
If you haven't, please take the damn box off-line *now* in the interest of good netizenship. Do whatever forensics seem prudent, off-line. At this point, nobody knows what is happening and this box needs to be offline until it is thoroughly secured.
The minimum forensics you need to do (or have done for you if you need help) is to determine where the attack came from and how it succeeded so you won't get caught with your knickers around your ankles again.
As soon as the attack vector is known, close it down on your other servers as quickly as you can.
Conventional wisdom is to cold load the compromised server before returning it to service, because the bad guys often leave multiple back doors. Fixing the attack point is not enough.
Regards, Ray
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack 22375 apache 15 0 964 560 472 S 0.3 0.0 0:04.21 atack 22858 apache 15 0 964 560 472 S 0.3 0.0 0:02.87 atack 22997 apache 15 0 964 560 472 S 0.3 0.0 0:04.11 atack 22999 apache 15 0 964 560 472 S 0.3 0.0 0:02.22 atack 23007 apache 15 0 964 560 472 S 0.3 0.0 0:03.79 atack 23099 apache 15 0 964 556 472 S 0.3 0.0 0:02.18 atack 23101 apache 15 0 964 556 472 S 0.3 0.0 0:02.48 atack 23108 apache 15 0 964 556 472 S 0.3 0.0 0:03.59 atack 23109 apache 15 0 964 556 472 S 0.3 0.0 0:02.75 atack 23112 apache 15 0 972 504 412 S 0.3 0.0 0:04.70 atack 23115 apache 15 0 964 556 472 S 0.3 0.0 0:03.75 atack 23116 apache 15 0 964 556 472 S 0.3 0.0 0:02.80 atack 23121 apache 15 0 972 504 412 S 0.3 0.0 0:03.79 atack 23384 apache 15 0 964 556 472 S 0.3 0.0 0:01.63 atack 23389 apache 15 0 964 556 472 S 0.3 0.0 0:03.52 atack 23392 apache 15 0 964 556 472 S 0.3 0.0 0:01.61 atack 23397 apache 15 0 964 556 472 S 0.3 0.0 0:01.62 atack 23405 apache 15 0 964 556 472 S 0.3 0.0 0:03.64 atack
When i 'ps -ef' i can see many lines as below;
apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100 apache 24292 23378 0 11:00 ? 00:00:01 ./atack 100 apache 24335 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24344 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24347 23378 0 11:02 ? 00:00:00 ./atack 100 apache 24358 23378 0 11:04 ? 00:00:00 ./atack 100
Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!
I good tool to have on your linux box that may help, some.
http://rkhunter.sourceforge.net/
http://rpmfind.net/linux/rpm2html/search.php?query=rkhunter
After installing do.
rkhunter --update
rkhunter -c
And see if it finds anything.
I usually watch and listen to this mailing list but this one really caught my eye.. I used to do alot of this in the military for 20yrs on nix boxes. Now I am a net engineer for a mid sized wisp. I have seen how brutal attacks take place on nix boxes. When I config a nix box the first thing I do is set the firewall up to block all ports above 1048 and only let in or out what ports are needed for the machine. My favorite ports to block are ftp,ssh and telnet. I will configure different ports for those apps if they are needed. I even block these common ports on our gateway to the network and only allow certain accounts inside the net access because they do not know how to change their ports to something uncommon. Most root kits are hard scripted for the common ports, unless the attacker is smart enough to use a port scanner try and find alternate ports but I can also block most scanners by dropping certain connection types. I have had a machine online for about 16yrs uptime with no attacks. They try but they die:) If it was easy enough for a root kit to get access to your machine then there are some definite holes in the system.
Matt wrote:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack 23479 apache 15 0 964 556 472 S 0.7 0.0 0:01.94 atack 22170 apache 15 0 964 560 472 S 0.3 0.0 0:05.23 atack 22375 apache 15 0 964 560 472 S 0.3 0.0 0:04.21 atack 22858 apache 15 0 964 560 472 S 0.3 0.0 0:02.87 atack 22997 apache 15 0 964 560 472 S 0.3 0.0 0:04.11 atack 22999 apache 15 0 964 560 472 S 0.3 0.0 0:02.22 atack 23007 apache 15 0 964 560 472 S 0.3 0.0 0:03.79 atack 23099 apache 15 0 964 556 472 S 0.3 0.0 0:02.18 atack 23101 apache 15 0 964 556 472 S 0.3 0.0 0:02.48 atack 23108 apache 15 0 964 556 472 S 0.3 0.0 0:03.59 atack 23109 apache 15 0 964 556 472 S 0.3 0.0 0:02.75 atack 23112 apache 15 0 972 504 412 S 0.3 0.0 0:04.70 atack 23115 apache 15 0 964 556 472 S 0.3 0.0 0:03.75 atack 23116 apache 15 0 964 556 472 S 0.3 0.0 0:02.80 atack 23121 apache 15 0 972 504 412 S 0.3 0.0 0:03.79 atack 23384 apache 15 0 964 556 472 S 0.3 0.0 0:01.63 atack 23389 apache 15 0 964 556 472 S 0.3 0.0 0:03.52 atack 23392 apache 15 0 964 556 472 S 0.3 0.0 0:01.61 atack 23397 apache 15 0 964 556 472 S 0.3 0.0 0:01.62 atack 23405 apache 15 0 964 556 472 S 0.3 0.0 0:03.64 atack When i 'ps -ef' i can see many lines as below; apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100 apache 24292 23378 0 11:00 ? 00:00:01 ./atack 100 apache 24335 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24344 23378 0 11:01 ? 00:00:00 ./atack 100 apache 24347 23378 0 11:02 ? 00:00:00 ./atack 100 apache 24358 23378 0 11:04 ? 00:00:00 ./atack 100 Hell, has my centos 5.3 box been hacked??? Help !!!!!!!!!!I good tool to have on your linux box that may help, some.
http://rkhunter.sourceforge.net/
http://rpmfind.net/linux/rpm2html/search.php?query=rkhunter
After installing do.
rkhunter --update
rkhunter -c
And see if it finds anything.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Matt, great idea.... I FOUND SOMETHING... pls see below...
________________________________
From: Matt lm7812@gmail.com To: CentOS mailing list centos@centos.org Sent: Thursday, June 4, 2009 4:40:57 AM Subject: Re: [CentOS] Centos 5.3 -> Apache - Under Attack ? Oh hell....
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 23119 apache 15 0 964 556 472 S 0.7 0.0 0:03.68 atack
When i 'ps -ef' i can see many lines as below;
apache 24253 23378 0 10:54 ? 00:00:00 ./atack 100 apache 24286 23378 0 10:59 ? 00:00:00 ./atack 100
I good tool to have on your linux box that may help, some. http://rkhunter.sourceforge.net/ http://rpmfind.net/linux/rpm2html/search.php?query=rkhunter After installing do. rkhunter --update rkhunter -c And see if it finds anything.
I DID FIND SOMETHING...NOT SURE WHAT THOUGH ;)
* Filesystem checks Checking /dev for suspicious files... [ OK ] Scanning for hidden files... [ Warning! ] --------------- /etc/.pwd.lock /usr/share/man/man1/..1.gz /dev/.udev --------------- Please inspect: /usr/share/man/man1/..1.gz (gzip compressed data, from Unix, max compression) /dev/.udev (directory)
The contents of the /dev/.udev folder;
drwxr-xr-x 2 root root 540 Jun 8 15:41 db drwxr-xr-x 2 root root 740 Jun 8 15:41 failed -rw-r--r-- 1 root root 4 Jun 8 15:42 uevent_seqnum
The contents of the ../man1/ folder ;
[root@fwg man1]# ls -al :.1.gz -rw-r--r-- 1 root root 40 Jan 22 09:14 :.1.gz
[root@fwgw man1]# ls -al [.1.gz -rw-r--r-- 1 root root 40 Jan 22 09:14 [.1.gz
Anything out of the ordinary?
---------------------------- Scan results ----------------------------
MD5 scan Skipped <--- WHY SKIPPED ? bcos OS unknown as shown in the NOTE below?
File scan Scanned files: 342 Possible infected files: 0
Application scan Vulnerable applications: 0
Scanning took 32 seconds
....................... end .........................................
NOTE: When we run rkhunter, rkhunter says the lines below...eventhough i installed frm the centos repo? but still it says its an unknown OS
Rootkit Hunter 1.2.9 is running Determining OS... Unknown Warning: This operating system is not fully supported! All MD5 checks will be skipped!
Anything out of the ordinary?
-
Anne Wilson -
Bob Hoffman -
bruce -
DAVID M -
Drew -
Geoff Galitz -
Ian Forde -
John R. Dennison -
JohnS -
Les Mikesell -
Linux Advocate -
Matt -
Michael A. Peters -
Neil Aggarwal -
Nicolas Thierry-Mieg -
Ralph Angenendt -
Raymond Lillard -
Ross Walker -
Scott Silva -
William L. Maltby -
William Warren