LAN-to-gateway traffic (e.g., a test FTP of a large file from the gateway to a machine on one of the LANs) begins to degrade as the LAN-to-internet traffic increases. That's not surprising, but it degrades disproportionately, i.e. when the FTP begins to show intermittent stalls, the total traffic visible at the router on the internet side of the gateway is only in the just-over-10Mb/s range.
Once we get to this point, no matter how many more LAN-to-internet connections become active, the router on the internet side never sees much over 10Mb/s of traffic. We're not losing data or having an unusual number of connection timeouts; each connection just slows down. We figured on some slowdown for NAT, but not 80%+.
LAN-to-LAN traffic that doesn't involve the gateway behaves more like we'd expect, but I'm not sure that eliminates the switch as the culprit.
Maybe it is time for some kernel networking tuning.
This will definetly require more memory, but should speed things up. This is on a CentOS 4 machine .. I don't have a CentOS 3 machine to test on.
Add the following lines to /etc/sysctl.conf
net.core.rmem_default = 67108864 net.core.wmem_default = 67108864 net.core.rmem_max = 67108864 net.core.wmem_max = 67108864 net.ipv4.tcp_mem = 4096 67108864 67108864 net.ipv4.tcp_rmem = 4096 67108864 67108864 net.ipv4.tcp_wmem = 4096 67108864 67108864 net.ipv4.ip_local_port_range = 32768 65535 net.ipv4.tcp_max_syn_backlog = 8192
After adding these lines, run "sysctl -p"
Hope this helps.
Barry