Hello everyone.
I've stumbled upon a strange networking issue with multiple interfaces on CentOS 5. The network setup is just like the diagram in http://lartc.org/howto/lartc.rpdb.multiple-links.html It looks like linux is not routing correctly outgoing packets on interfaces different from the one of the default gateway, but instead broadcasts an ARP request on the link, looking for the destination host, which doesn't make any sense as the remote host is far away. I suspect there's a bug in the kernel in the routing decision algorithm, as the issue is absent in kernel releases up to 2.6.18-194.8.1.el5 (CentOS 5.5), while I've tracked it down at least from 2.6.18-308.1.1.el5 release onwards (CentOS 5.8). I'm not sure in which release inbetween appeared first.
I'll try to explain with an example. Let's say that my IF1 is eth1 with 100.10.10.1/24 and gateway 100.10.10.254 and IF2 is eth2 99.10.11.1/24 with gateway 99.10.11.254. Inside network is eth0 with 192.168.0.1/24. My main routing table looks something like
# route -n Destination Gateway Genmask Flags Metric Ref Use Iface 99.10.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 100.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 0.0.0.0 100.10.10.254 0.0.0.0 UG 0 0 0 eth1
Then I've setup two additional routing tables with iproute2 and linked to the main rule list like this
# cat >> /etc/iproute2/rt_tables << EOF 10 provider1 20 provider2 EOF
# ip route add 100.10.10.0/24 dev eth1 table provider1 # ip route add default via 100.10.10.254 dev eth1 table provider1 # ip rule add from 100.10.10.0/24 table provider1
# ip route add 99.10.11.0/24 dev eth2 table provider2 # ip route add default via 99.10.11.254 dev eth2 table provider2 # ip rule add from 99.10.11.0/24 table provider2
So the result is the following:
# ip rule list 0: from all lookup 255 32764: from 99.10.11.0/24 lookup provider2 32765: from 100.10.10.0/24 lookup provider1 32766: from all lookup main 32767: from all lookup default
# ip route list table provider1 100.10.10.0/24 dev eth1 scope link default via 100.10.10.254 dev eth1
# ip route list table provider2 99.10.11.0/24 dev eth2 scope link default via 99.10.11.254 dev eth2
Now the issue. With kernel 2.6.18-194.8.1.el5, if I try two different ping forcing outgoing interface everything looks good:
# ping -c 3 -I eth1 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 100.10.10.1 eth1: 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=49 time=16.9 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=49 time=19.4 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=49 time=20.7 ms
--- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 16.378/16.796/17.289/0.390 ms
and
# ping -c 3 -I eth2 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 99.10.11.1 eth2: 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=48 time=24.9 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=48 time=25.3 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=48 time=24.5 ms
--- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2002ms rtt min/avg/max/mdev = 16.378/16.796/17.289/0.390 ms
If I repeat the ping, but with a newer kernel, let's say 2.6.18-308.1.1.el5, the result is the following:
# ping -c 3 -I eth1 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 100.10.10.1 eth1: 56(84) bytes of data. 64 bytes from 8.8.8.8: icmp_seq=1 ttl=48 time=36.8 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=48 time=36.8 ms 64 bytes from 8.8.8.8: icmp_seq=3 ttl=48 time=35.5 ms
--- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2000ms rtt min/avg/max/mdev = 35.542/36.433/36.887/0.630 ms
# ping -c 3 -I eth2 8.8.8.8 PING 8.8.8.8 (8.8.8.8) from 99.10.11.1 eth2: 56(84) bytes of data.
From 99.10.11.1 icmp_seq=1 Destination Host Unreachable From 99.10.11.1 icmp_seq=2 Destination Host Unreachable From 99.10.11.1 icmp_seq=3 Destination Host Unreachable
--- 8.8.8.8 ping statistics --- 3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2001ms , pipe 3
I was puzzled by this behaviour, so I traced network traffic with tcpdump, and the output is the following:
# tcpdump -n -i eth2 src or dst 8.8.8.8 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth2, link-type EN10MB (Ethernet), capture size 96 bytes 10:03:34.951676 arp who-has 8.8.8.8 tell 99.10.11.1 10:03:35.951719 arp who-has 8.8.8.8 tell 99.10.11.1 10:03:36.951662 arp who-has 8.8.8.8 tell 99.10.11.1
3 packets captured 3 packets received by filter 0 packets dropped by kernel
It looks like the routing algorithm discards advanced routing, forgetting about the second default gateway on provider2 table.
Adding a second default route to the main routing table seems to resolve the issue, but it wasn't necessary before and I'm not sure is a correct solution.
# route add default gw 100.10.11.254 metric 100 # route -n Destination Gateway Genmask Flags Metric Ref Use Iface 99.10.11.0 0.0.0.0 255.255.255.0 U 0 0 0 eth2 100.10.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1 192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0 0.0.0.0 99.10.11.254 0.0.0.0 UG 100 0 0 eth2 0.0.0.0 100.10.10.254 0.0.0.0 UG 0 0 0 eth1
Any comments ?