My guys,
My firewall seems to block an attack my Centos / Sendmail boxes on port 110. These servers require a reboot after each attack. My firewall says it's blocked? Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
11/20/2008 02:53:04.864 - SYN flood attack dropped - 75.2.205.141, 48102 - 10.80.80.210, 110
11/20/2008 03:08:04.864 - SYN flood attack dropped - 75.2.205.141, 64955, greatcooks.biz - 10.80.80.220, 110
11/20/2008 03:23:08.864 - SYN flood attack dropped - 75.2.205.141, 43068, greatcooks.biz - 10.80.80.210, 110
Any input would be much appreciated.
Thanks.
Chris Heiner wrote on Thu, 20 Nov 2008 08:48:50 -0800:
My firewall seems to block an attack my Centos / Sendmail boxes on port 110.
port 110 is your POP server, probably dovecot.
These servers require a reboot after each attack.
Because of what?
My firewall says it's blocked?
I don't see this statement in your logs. How/where does it say this?
Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
SYN floods are not unusual, even if it is not an attack. What or if you want to do something depends on your situation.
Kai
What would you like to know about my situation? I have 6 servers running Centos 4.x and every time I get a SYD flood on port 110 the servers require a reboot (all of them). Its been going on for a few months.
I have blocked the first few IP's but now its random, every few weeks.
Its only my Centos boxes as I have others that are not affect by it.
Does the help?
Thanks in advance.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Kai Schaetzl Sent: Thursday, November 20, 2008 11:31 AM To: centos@centos.org Subject: Re: [CentOS] SYD flood dropped on Sendmail (centos 4.x)
Chris Heiner wrote on Thu, 20 Nov 2008 08:48:50 -0800:
My firewall seems to block an attack my Centos / Sendmail boxes on port
110.
port 110 is your POP server, probably dovecot.
These servers require a reboot after each attack.
Because of what?
My firewall says it's blocked?
I don't see this statement in your logs. How/where does it say this?
Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
SYN floods are not unusual, even if it is not an attack. What or if you want to do something depends on your situation.
Kai
Chris, you still didn't answer *why* you have to reboot them. What exactly is the symptom that makes you think you have to reboot?
I assume now that with "My firewall says it's blocked" you referred to the drops? (Next time say so, as this wording is really ambiguous.)
What would you like to know about my situation? I have 6 servers running
Yeah, so you are not a home user where one could rate-limit the port ;-)
Kai
I get complaints about "the servers asking for username and password". I started test@ accounts all many servers to try and track it down. And it happens to all the servers that receive a SYN Flood. I.E. the problem with each server co insides with firewall logs. Its a pattern every few weeks, sometimes a few servers sometimes 2 or 3 but it always matches up with the firewall log. I now have emails sent to me to alert of a port 110 SYD flood so I am aware of the problem before I get a full voicemail box from complaints. Most of the time it's in the middle of the night at 2am to 3am and the problem is resolved by start of business day. So that would rule out heavy usage from my users as the network reports show that it's quiet. We have 10 MB fiber connection and all traffic is logged at many levels.
I have tried restarting POP and SMTP in the past, but rebooting seems to work and if there isnt a fix I will have to continue this as I have many other networking issues to resolve.
I just thought I would throw this problem out to the group and see if anyone has any good ideas.
I have tracked this mail list for years and everyone is extremely knowledgeable.
Thanks for any replies..
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Kai Schaetzl Sent: Thursday, November 20, 2008 12:53 PM To: centos@centos.org Subject: Re: [CentOS] SYD flood dropped on Sendmail (centos 4.x)
Chris, you still didn't answer *why* you have to reboot them. What exactly is the symptom that makes you think you have to reboot?
I assume now that with "My firewall says it's blocked" you referred to the drops? (Next time say so, as this wording is really ambiguous.)
What would you like to know about my situation? I have 6 servers running
Yeah, so you are not a home user where one could rate-limit the port ;-)
Kai
Hi Chris,
You still did not give enough detail of what happens on the machine when the problem strikes you. For instance:
- What is in /var/log/messages at that time? - What is in "dmesg" output? - What is in the log of your POP3 server (you still did not tell which program you are using)? - What happens if you run "telnet localhost 110" or "telnet <hostname> 110" (replacing <hostname> with the real host name) while connected to the problematic host? - What happens if you run "telnet <hostname> 110" from another machine?
You can also try to run POP3 commands in the telnet session to diagnose what is the error message when the password is refused.
If you want help, you should start by giving more information about the problem. You haven't even said which version of CentOS you are running, if you're fully updated or not, and the program you refer to (sendmail) clearly has nothing to do with the issue on your firewall since that one is reporting port 110 which is POP3.
Also, just because the logs of your firewall say something, that does not mean it's the root of your problem. Maybe that could be an effect of something else that is happening on your machine and causing packets to be refused in a way that would trick the firewall to think there's an attack.
HTH, Filipe
Chris Heiner wrote on Thu, 20 Nov 2008 13:43:44 -0800:
I get complaints about "the servers asking for username and password".
from your users or what? Of course, they may complain. A big dictionary attack can take almost all the bandwidth for some time or leave a backlog of dovecot instances. Please, as I understand you are a server adminstrator for quite a few machines, correct? Yet, you are answering in a way as if you just brought your first server online.
Btw, it's a *SYN* flood, not a SYD flood and that won't change even if you repeat it again and again.
I
started test@ accounts all many servers to try and track it down.
Pardon, you did what?
I have tried restarting POP and SMTP in the past
You may want to kill all dovecot instances, in case you *are* running dovecot (if not, then of what you use, but I know that dovecot likes to hang in this way if hammered). Just restarting it may not kill the backlog of hanging connections. A "ps ax|grep login" would help to see if instances are still running. Restarting SMTP: again, this has nothing to do with SMTP!
Kai
on 11-20-2008 3:31 PM Kai Schaetzl spake the following:
Chris Heiner wrote on Thu, 20 Nov 2008 13:43:44 -0800:
I get complaints about "the servers asking for username and password".
from your users or what? Of course, they may complain. A big dictionary attack can take almost all the bandwidth for some time or leave a backlog of dovecot instances. Please, as I understand you are a server adminstrator for quite a few machines, correct? Yet, you are answering in a way as if you just brought your first server online.
Btw, it's a *SYN* flood, not a SYD flood and that won't change even if you repeat it again and again.
I
started test@ accounts all many servers to try and track it down.
Pardon, you did what?
I have tried restarting POP and SMTP in the past
You may want to kill all dovecot instances, in case you *are* running dovecot (if not, then of what you use, but I know that dovecot likes to hang in this way if hammered). Just restarting it may not kill the backlog of hanging connections. A "ps ax|grep login" would help to see if instances are still running. Restarting SMTP: again, this has nothing to do with SMTP!
Kai
CentOS 4 comes with a very OLD version of dovecot. If you are using dovecot, you can get a much newer version at atrpms.net. The upgrade might be all you need to fix it.
At 07:03 PM 11/20/2008, you wrote:
on 11-20-2008 3:31 PM Kai Schaetzl spake the following:
Chris Heiner wrote on Thu, 20 Nov 2008 13:43:44 -0800:
I get complaints about "the servers asking for username and password".
from your users or what? Of course, they may complain. A big dictionary attack can take almost all the bandwidth for some time or leave a backlog of dovecot instances. Please, as I understand you are a server adminstrator for quite a few machines, correct? Yet, you are answering in a way as if you just brought your first server online.
Btw, it's a *SYN* flood, not a SYD flood and that won't change even if you repeat it again and again.
I
started test@ accounts all many servers to try and track it down.
Pardon, you did what?
I have tried restarting POP and SMTP in the past
You may want to kill all dovecot instances, in case you *are* running dovecot (if not, then of what you use, but I know that dovecot likes to hang in this way if hammered). Just restarting it may not kill the backlog of hanging connections. A "ps ax|grep login" would help to see if instances are still running. Restarting SMTP: again, this has nothing to do with SMTP!
Kai
CentOS 4 comes with a very OLD version of dovecot. If you are using dovecot, you can get a much newer version at atrpms.net. The upgrade might be all you need to fix it.
Watch out for this gotcha! The Dovecot version 1.0.x that comes with CentOS 5.x is much better and I run it and would recommend it, but the configs for 0.99.x (Came with CentOS 4.x) are incompatible with the previous version.
Cheers, Glenn
Good advice!
Thanks for helping without the "corrective elitist attitude"!
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Glenn Sent: Thursday, November 20, 2008 5:16 PM To: CentOS mailing list Subject: Re: [CentOS] Re: SYD flood dropped on Sendmail (centos 4.x)
At 07:03 PM 11/20/2008, you wrote:
on 11-20-2008 3:31 PM Kai Schaetzl spake the following:
Chris Heiner wrote on Thu, 20 Nov 2008 13:43:44 -0800:
I get complaints about "the servers asking for username and password".
from your users or what? Of course, they may complain. A big dictionary attack can take almost all the bandwidth for some time or leave a
backlog
of dovecot instances. Please, as I understand you are a server adminstrator for quite a few machines, correct? Yet, you are answering in a way as if you just
brought
your first server online.
Btw, it's a *SYN* flood, not a SYD flood and that won't change even if
you
repeat it again and again.
I
started test@ accounts all many servers to try and track it down.
Pardon, you did what?
I have tried restarting POP and SMTP in the past
You may want to kill all dovecot instances, in case you *are* running dovecot (if not, then of what you use, but I know that dovecot likes to hang in this way if hammered). Just restarting it may not kill the
backlog
of hanging connections. A "ps ax|grep login" would help to see if instances are still running. Restarting SMTP: again, this has nothing to do with SMTP!
Kai
CentOS 4 comes with a very OLD version of dovecot. If you are using dovecot, you can get a much newer version at atrpms.net. The upgrade might be all you need to fix it.
Watch out for this gotcha! The Dovecot version 1.0.x that comes with CentOS 5.x is much better and I run it and would recommend it, but the configs for 0.99.x (Came with CentOS 4.x) are incompatible with the previous version.
Cheers, Glenn
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
---------------------------------------------- Gateway Anti-Spam Anti-Virus Protection by Network Designs Inc. 949-727-3393 For a complete list of services go to www.networkdesignsinc.com ----------------------------------------------
Scott Silva wrote on Thu, 20 Nov 2008 16:03:04 -0800:
CentOS 4 comes with a very OLD version of dovecot. If you are using dovecot, you can get a much newer version at atrpms.net. The upgrade might be all you need to fix it.
The dovecot in CentOS 5 exhibits the same problem when hammered by dictionary attacks. Is the atrpms version newer?
Kai
on 11-20-2008 5:31 PM Kai Schaetzl spake the following:
Scott Silva wrote on Thu, 20 Nov 2008 16:03:04 -0800:
CentOS 4 comes with a very OLD version of dovecot. If you are using dovecot, you can get a much newer version at atrpms.net. The upgrade might be all you need to fix it.
The dovecot in CentOS 5 exhibits the same problem when hammered by dictionary attacks. Is the atrpms version newer?
Kai
You can get 1.0.15 which is the recent stable for the 1.0 series, and you can get 1.1.16 which has many new improvements over 1.0, and is the current stable branch. I think the 1.1 branch has some changes to the auth code that might help. Read the dovecot wiki for the steps you need to follow to upgrade, especially if you want to go back.
I really recommend you at least go to the 1.0 branch instead of the 0.99 beta in CentOS 4. The indexing improvements alone are worth it.
on 11-21-2008 11:53 AM Scott Silva spake the following:
on 11-20-2008 5:31 PM Kai Schaetzl spake the following:
Scott Silva wrote on Thu, 20 Nov 2008 16:03:04 -0800:
CentOS 4 comes with a very OLD version of dovecot. If you are using dovecot, you can get a much newer version at atrpms.net. The upgrade might be all you need to fix it.
The dovecot in CentOS 5 exhibits the same problem when hammered by dictionary attacks. Is the atrpms version newer?
Kai
You can get 1.0.15 which is the recent stable for the 1.0 series, and you can get 1.1.16 which has many new improvements over 1.0, and is the current stable branch. I think the 1.1 branch has some changes to the auth code that might help. Read the dovecot wiki for the steps you need to follow to upgrade, especially if you want to go back.
I really recommend you at least go to the 1.0 branch instead of the 0.99 beta in CentOS 4. The indexing improvements alone are worth it.
Another option is something like fail2ban, and have it drop the connections and add a firewall rule when you get too many bad attempts on that port. Fail2ban can read the logs and act for you before it gets too bad.
Good advice!
I will upgrade the Dovecot as it sounds like a good idea. I was also considering just redirecting the inbound port from 110 to another port.
Your simple answer is much appreciated.
Thanks for helping without the "corrective elitist attitude"!
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Scott Silva Sent: Thursday, November 20, 2008 4:03 PM To: centos@centos.org Subject: [CentOS] Re: SYD flood dropped on Sendmail (centos 4.x)
on 11-20-2008 3:31 PM Kai Schaetzl spake the following:
Chris Heiner wrote on Thu, 20 Nov 2008 13:43:44 -0800:
I get complaints about "the servers asking for username and password".
from your users or what? Of course, they may complain. A big dictionary attack can take almost all the bandwidth for some time or leave a backlog of dovecot instances. Please, as I understand you are a server adminstrator for quite a few machines, correct? Yet, you are answering in a way as if you just brought your first server online.
Btw, it's a *SYN* flood, not a SYD flood and that won't change even if you
repeat it again and again.
I
started test@ accounts all many servers to try and track it down.
Pardon, you did what?
I have tried restarting POP and SMTP in the past
You may want to kill all dovecot instances, in case you *are* running dovecot (if not, then of what you use, but I know that dovecot likes to hang in this way if hammered). Just restarting it may not kill the backlog
of hanging connections. A "ps ax|grep login" would help to see if instances are still running. Restarting SMTP: again, this has nothing to do with SMTP!
Kai
CentOS 4 comes with a very OLD version of dovecot. If you are using dovecot, you can get a much newer version at atrpms.net. The upgrade might be all you need to fix it.
11/20/2008 02:53:04.864 - SYN flood attack dropped - 75.2.205.141, 48102 - 10.80.80.210, 110
11/20/2008 03:08:04.864 - SYN flood attack dropped - 75.2.205.141, 64955, greatcooks.biz - 10.80.80.220, 110
11/20/2008 03:23:08.864 - SYN flood attack dropped - 75.2.205.141, 43068, greatcooks.biz - 10.80.80.210, 110
These are the statements from my Firewall saying that it was dropped.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Kai Schaetzl Sent: Thursday, November 20, 2008 11:31 AM To: centos@centos.org Subject: Re: [CentOS] SYD flood dropped on Sendmail (centos 4.x)
Chris Heiner wrote on Thu, 20 Nov 2008 08:48:50 -0800:
My firewall seems to block an attack my Centos / Sendmail boxes on port
110.
port 110 is your POP server, probably dovecot.
These servers require a reboot after each attack.
Because of what?
My firewall says it's blocked?
I don't see this statement in your logs. How/where does it say this?
Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
SYN floods are not unusual, even if it is not an attack. What or if you want to do something depends on your situation.
Kai
Kai Schaetzl wrote:
Chris Heiner wrote on Thu, 20 Nov 2008 08:48:50 -0800:
My firewall seems to block an attack my Centos / Sendmail boxes on port 110.
port 110 is your POP server, probably dovecot.
These servers require a reboot after each attack.
Because of what?
My firewall says it's blocked?
I don't see this statement in your logs. How/where does it say this?
Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
SYN floods are not unusual, even if it is not an attack. What or if you want to do something depends on your situation.
If you have a popular server you can get what appear to be syn floods from broken asymmetrical routing or bad firewall settings that permit what would ordinarily be a normal number of client connection requests to reach you but keep your response from getting back. So the clients sit and retry, hammering you with syn's.
Les,
I have had that issue before with high traffic users and you are correct, but I think this may be another issue as the its an off hours issue.
Thanks
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Les Mikesell Sent: Thursday, November 20, 2008 12:07 PM To: CentOS mailing list Subject: Re: [CentOS] SYD flood dropped on Sendmail (centos 4.x)
Kai Schaetzl wrote:
Chris Heiner wrote on Thu, 20 Nov 2008 08:48:50 -0800:
My firewall seems to block an attack my Centos / Sendmail boxes on port
110.
port 110 is your POP server, probably dovecot.
These servers require a reboot after each attack.
Because of what?
My firewall says it's blocked?
I don't see this statement in your logs. How/where does it say this?
Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
SYN floods are not unusual, even if it is not an attack. What or if you want to do something depends on your situation.
If you have a popular server you can get what appear to be syn floods from broken asymmetrical routing or bad firewall settings that permit what would ordinarily be a normal number of client connection requests to reach you but keep your response from getting back. So the clients sit and retry, hammering you with syn's.
Chris Heiner wrote:
My guys,
My firewall seems to block an attack my Centos / Sendmail boxes on port 110. These servers require a reboot after each attack. My firewall says it’s blocked? Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
11/20/2008 02:53:04.864 - SYN flood attack dropped - 75.2.205.141, 48102 - 10.80.80.210, 110
11/20/2008 03:08:04.864 - SYN flood attack dropped - 75.2.205.141, 64955, greatcooks.biz - 10.80.80.220, 110
11/20/2008 03:23:08.864 - SYN flood attack dropped - 75.2.205.141, 43068, greatcooks.biz - 10.80.80.210, 110
Any input would be much appreciated.
Thanks.
If these are to bogus email addresses, you might try letting sendmail itself throttle the attacks. Look into sendmail's BAD_RCPT_THROTTLE. This has done wonders for my systems.
John Hinton
John Hinton wrote:
Chris Heiner wrote:
My guys,
My firewall seems to block an attack my Centos / Sendmail boxes on port 110. These servers require a reboot after each attack. My firewall says it’s blocked? Do I need to patch something on sendmail? Or is my firewall not doing its job (Sonicwall)? This is not the first time this has happened.
11/20/2008 02:53:04.864 - SYN flood attack dropped - 75.2.205.141, 48102 - 10.80.80.210, 110
11/20/2008 03:08:04.864 - SYN flood attack dropped - 75.2.205.141, 64955, greatcooks.biz - 10.80.80.220, 110
11/20/2008 03:23:08.864 - SYN flood attack dropped - 75.2.205.141, 43068, greatcooks.biz - 10.80.80.210, 110
Any input would be much appreciated.
Thanks.
If these are to bogus email addresses, you might try letting sendmail itself throttle the attacks. Look into sendmail's BAD_RCPT_THROTTLE. This has done wonders for my systems.
John Hinton
Duh... obviously I can't read. Sorry.
John Hinton
-
Chris Heiner -
Filipe Brandenburger -
Glenn -
John Hinton -
Kai Schaetzl -
Les Mikesell -
Scott Silva