John Doe wrote: > Hi, > > I was asked to check some TIME_WAITs "problems" (my boss thinks there should almost never be any) and I bumped into something strange... > I SHOULD be able to answer this, I was involved when we solved the PANIX TCP-WAIT attack way back when... But the OS has changed since then, and I don't work with internals. > All of our servers have apparently normal (in my opinion) 60s TIME_WAITs (even if it strangely caps around 14000 in my tests)... > But one of them behaves differently (and my boss thinks it is the normal behavior). > If I make 10000 rapid connections/selects/deconnections to mysql on this server, I get like 1 TW after around 3000, another TW around 6000 and another TW around 9000... That makes 3 TWs only. And they last 60 seconds... > I am told this server was not really setup differently (no custom kernel). > All servers are CentOS 5.2, kernels 2.6.18-92.1.[22|10].el5. > I compared the values in sysctl.conf and /proc/sys/net/ipv4/* and nothing different. > > So, am I correct in thinking that seeing thousands TWs when there was a burst of thousands connections is normal? > Any idea why so few TWs on this server? Any conf file I should check? > In your testing is the source IP the same for all with just different source port? Or are you varying your source IP as well? I don't know what spoofing smarts are in the kernel to detect SYN/ACK attacks. Are you running Shorewall or any similar tool that will detect SYN/ACK attacks and might be seeing this 'test' as an attack to limit? > # cat /proc/sys/net/ipv4/tcp_fin_timeout > 60 > # cat /proc/sys/net/ipv4/tcp_max_tw_buckets > 180000 > I remember when this was 256. And that on a 'high-end' SUN, AIX, or SGI server! With that many buckets it will take a lot of TW before random discard kicks in. I wonder what the threshhold is? > # cat /proc/sys/net/ipv4/tcp_tw_recycle > 0 > # cat /proc/sys/net/ipv4/tcp_tw_reuse > 0 > > When I googled for it, many people were pointing to the tcp_fin_timeout value ... Is it really related to TWs? Well, yes. How long do you let a TW sit around waiting for a proper FIN or even a RST? Read the TCP RFC as to why there is a TW in the state machine. Boy has it been years since I cracked that one open... It is really a resource issue. If you have gobs of memory, what do you care if 20% of your TCBs are tied up in TW? Or if 50% are tied up? Eventually they close out. There are CPU requirements to track all of these as well, but these tasks tend to run using idle time. If you are seeing memory or cpu related bottlenecks, tools like Shorewall can block rapid connects.