[CentOS] TIME_WAITs...

Mon Mar 2 15:40:43 UTC 2009
John Doe <jdmls at yahoo.com>

> I spent a bunch of time researching TIME_WAIT on linux and didn't
> find much useful information. There's a couple kernel parameters
> to change the settings though the only docs for them that I could
> find say don't touch them unless you REALLY know what your doing

Only things I found are the hardcoded values in include/net/tcp.h:

  #define TCP_TIMEWAIT_LEN (60*HZ) /* how long to wait to destroy TIME-WAIT
                                  * state, about 60 seconds     */
                                 /* BSD style FIN_WAIT2 deadlock breaker.
                                  * It used to be 3min, new value is 60sec,
                                  * to combine FIN-WAIT-2 timeout with
                                  * TIME-WAIT timer.

Our "issue" is on the LAN side: front servers connecting to the dbs.
So I wonder if 60s is not too long for the delayed packets problem, when the sources and the targets are one gigabit switch away...

> The app that runs on that box is very high volume, so we get a large
> number of TIME_WAITs, during performance testing on a dual proc quad
> core we can get up to 63,000 of them.

Hum... I think I just understood why I cap around 14,000 in my tests...
cat /proc/sys/net/ipv4/ip_local_port_range 
32768   61000
(61000-32768)/2 = 14116
Could that be it?

> So IMO don't worry about time waits unless your seriously in the
> 10s of thousands, at which point you may want to think about
> optimizing the traffic flow to your systems like we did with
> our load balancers.

We already use LVS+keepalived and it seems to work fine so far (except when I tested 1.1.16 ^_^).