[CentOS] iptables starts blocking outbound http traffic

Thu Nov 6 16:04:41 UTC 2008
Filipe Brandenburger <filbranden at gmail.com>

Hi,

On Thu, Nov 6, 2008 at 10:42, Neil Aggarwal <neil at jammconsulting.com> wrote:
> If this is the source of the problem, how would restarting
> httpd and tomcat help?  I did not restart the machine nor
> reset iptables.

Because this might potentially close several connections and free
slots in the conntrack table.

You are right that your conntrack table size is high enough and this
should not be happening. It might be an attack, a synflood or
something, that is causing this problem to happen. In that case, the
semi-opened connections will be kept on the table, but as the other
side will not complete the handshake, they will only be removed from
the table after a timeout. I also think that when you stop Apache,
there will be no process listening on port 80 anymore, and then
conntrack may get rid of those semi-opened connections since the other
side is not listening anymore. A lot of especulation here, but it
might be what is affecting you.

In any case, next time you have this same problem, considering looking
at the counters to see if _count is reaching _max, that would confirm
the hypothesis.

If that is indeed the case, you can dump the conntrack information
with this command:
# cat /proc/net/ip_conntrack

You can do that and save it to another file, restart Apache and do the
same, so that you can see what is really happening there. This might
give you a better idea of why it's happening.

If conntrack is really overflowing, you may consider increasing the
table size, but this will mean more memory usage on your server.

Alternatively you might choose to redo your firewall rules to be
stateless, by removing --state NEW and --state ESTABLISHED, and by
adding ! --syn on the ones you want to allow for established
connections only. It's not going to be as perfect as actually tracking
the connections, but for protocols like HTTP is a good enough
compromise.

But your problem is probably being caused by something else, like an
attack, so probably the best way to deal with it is to find out what
is causing it and try to take measures to correct that problem
instead.

> I am not asking this to be argumentative, just trying to
> understand how the facts I am seeing are related.

No problem! Didn't sound argumentative to me in any way.

Let us know how that goes, and if you get more cluse, let us know if
you need more help in fixing the root problem.

HTH,
Filipe