Hello,

This is a CentOS 5.2 box configured as a router for a network handling about 200-300 Mbps, routing traffic to/from the internet for about 6,000 IPs.

After about 2-3 days, the kernel complains about "dst cache overflow" and even thought it hasn't crashed, the network is un-responsive. All IP forwarding stops and the server cannot be reached from any network interfaces.

After diagnosing the issue, it appears that there is a rt_cache leak in the kernel. At 5 minute intervals I collect the following two values:

`/sbin/ip -o route ls cache | wc -l` `grep ip_dst_cache /proc/slabinfo | awk -F' ' '{ print $2; }'`

The first value represents the number of cached routes in the network stack. The second value represents the number of cached route objects the kernel has allocated.

After collecting the data for 8 hours, I have seen the cached routes count remain fairly constant, but the number of cached route objects increase from about 220,000 objects to 410,000 objects. The cached routes count remains between 4,000 and 8,000 routes.

I posted a pretty graph at http://www.pier-pro.com/ip_dst_cache_leak.png (the blue line is the value of ip_dst_cache, the green value represents the count of cached routes).

Once the ip_dst_cache value reaches the value of /proc/sys/net/ipv4/route/max_size then the network fails and the kernel complains about 'dst cache overflow' whenever a packet arrives.

The only solution at that point is to perform a reboot.

Flushing the routing tables by `echo 1 > /proc/sys/net/ipv4/route/flush` only clears the cached routes, however the value of ip_dst_cache does not change.

According to http://linux.derkeiler.com/Mailing-Lists/Fedora/2005-07/1175.html this is a known bug that was fixed in 2.6.11, however, I'm running 2.6.18 (as updated with `yum update`)

I downloaded the kernel sources, and indeed, the kernel source contains the bug fix in the above article.

Therefore ... I'm at a loss as to where to go from here. Certainly rebooting the server every day is not an option, and increasing the max_size will just delay it.

Suggestions?

Thank you,

Hector