Hi,
On Thu, Nov 6, 2008 at 10:42, Neil Aggarwal neil@jammconsulting.com wrote:
If this is the source of the problem, how would restarting httpd and tomcat help? I did not restart the machine nor reset iptables.
Because this might potentially close several connections and free slots in the conntrack table.
You are right that your conntrack table size is high enough and this should not be happening. It might be an attack, a synflood or something, that is causing this problem to happen. In that case, the semi-opened connections will be kept on the table, but as the other side will not complete the handshake, they will only be removed from the table after a timeout. I also think that when you stop Apache, there will be no process listening on port 80 anymore, and then conntrack may get rid of those semi-opened connections since the other side is not listening anymore. A lot of especulation here, but it might be what is affecting you.
In any case, next time you have this same problem, considering looking at the counters to see if _count is reaching _max, that would confirm the hypothesis.
If that is indeed the case, you can dump the conntrack information with this command: # cat /proc/net/ip_conntrack
You can do that and save it to another file, restart Apache and do the same, so that you can see what is really happening there. This might give you a better idea of why it's happening.
If conntrack is really overflowing, you may consider increasing the table size, but this will mean more memory usage on your server.
Alternatively you might choose to redo your firewall rules to be stateless, by removing --state NEW and --state ESTABLISHED, and by adding ! --syn on the ones you want to allow for established connections only. It's not going to be as perfect as actually tracking the connections, but for protocols like HTTP is a good enough compromise.
But your problem is probably being caused by something else, like an attack, so probably the best way to deal with it is to find out what is causing it and try to take measures to correct that problem instead.
I am not asking this to be argumentative, just trying to understand how the facts I am seeing are related.
No problem! Didn't sound argumentative to me in any way.
Let us know how that goes, and if you get more cluse, let us know if you need more help in fixing the root problem.
HTH, Filipe