Hello,
This is on Centos 6 and not something I think is wrong with Centos 6 but I am looking to see if anybody else has experienced this and if there is solution. So thanks up front for indulging me.
Because Linux makes routing decisions before SNAT it is causing problems when trying to use FTP with two upstream providers in a load balanced setup.
Other than ftp, things seem to work OK. Below is my setup and tcpdump output that shows ftp packets trying to go out the wrong interface.
ip ru sh 0: from all lookup local 200: from y.y.y.174 lookup t1 201: from x.x.x.217 lookup t2 32766: from all lookup main 32767: from all lookup default
ip r s y.y.y.129 dev eth1 scope link 172.16.0.0/29 dev gre1 proto kernel scope link src 172.16.0.1 y.y.y.128/25 dev eth1 proto kernel scope link src y.y.y.174 10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.90 192.168.198.0/24 dev eth0 proto kernel scope link src 192.168.198.92 x.x.x.0/24 dev eth2 proto kernel scope link src x.x.x.217 10.0.128.0/17 dev eth0 proto kernel scope link src 10.0.129.88 default nexthop via y.y.y.129 dev eth1 weight 1 nexthop via x.x.x.1 dev eth2 weight 1
ip r s tab t1 default via y.y.y.129 dev eth1 src y.y.y.174
ip r s tab t2 default via x.x.x.1 dev eth2 src x.x.x.217
Chain PREROUTING (policy ACCEPT 1050K packets, 128M bytes) pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 423K packets, 35M bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT all -- * eth1 10.0.1.0/24 10.0.0.0/8 0 0 ACCEPT all -- * eth1 10.0.1.0/24 172.16.0.0/12 0 0 ACCEPT all -- * eth1 10.0.1.0/24 192.168.0.0/16 58 3480 SNAT all -- * eth1 10.0.1.0/24 0.0.0.0/0 to:y.y.y.174 4 240 SNAT all -- * eth2 10.0.1.0/24 0.0.0.0/0 to:x.x.x.217
lsmod | grep nf_ nf_conntrack_ipv6 7207 3 nf_defrag_ipv6 9873 1 nf_conntrack_ipv6 nf_nat_ftp 2602 0 nf_nat 18580 2 iptable_nat,nf_nat_ftp nf_conntrack_ipv4 7694 6 iptable_nat,nf_nat nf_defrag_ipv4 1039 1 nf_conntrack_ipv4 nf_conntrack_ftp 10475 1 nf_nat_ftp nf_conntrack 65524 7 iptable_nat,nf_conntrack_ipv6,xt_state,nf_nat_ftp,nf_nat,nf_conntrack_ipv4,nf_conntrack_ftp ipv6 264769 41 nf_conntrack_ipv6,nf_defrag_ipv6,ip6table_mangle,ip6_tunnel,tunnel6
connection starts out eth2 - but then subsequent packets that should be routed out eth2 are being sent out eth1 see below. eth2 x.x.x.217 tcpdump -nli eth2 host 131.247.254.5
16:23:06.062451 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [S], seq 1482565198, win 5840, options [mss 1460,sackOK,TS val 423546705 ecr 0,nop,wscale 6], length 0 16:23:06.076788 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [S.], seq 740341460, ack 1482565199, win 5792, options [mss 1460,sackOK,TS val 2565444838 ecr 423546705,nop,wscale 7], length 0 16:23:06.077224 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423546720 ecr 2565444838], length 0 16:23:06.096900 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565444858 ecr 423546720], length 96 16:23:06.316866 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565445077 ecr 423546720], length 96 16:23:06.764410 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565445515 ecr 423546720], length 96 16:23:07.634411 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565446391 ecr 423546720], length 96 16:23:09.394482 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565448143 ecr 423546720], length 96 16:23:12.886997 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565451647 ecr 423546720], length 96 16:23:19.892082 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565458655 ecr 423546720], length 96 16:23:33.907303 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565472671 ecr 423546720], length 96 16:24:01.935273 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565500703 ecr 423546720], length 96 16:24:57.993631 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565556767 ecr 423546720], length 96 16:26:50.107951 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [P.], seq 1:97, ack 1, win 46, options [nop,nop,TS val 2565668895 ecr 423546720], length 96 16:28:06.104031 IP 131.247.254.5.ftp > x.x.x.217.53651: Flags [FP.], seq 97:111, ack 1, win 46, options [nop,nop,TS val 2565744900 ecr 423546720], length 14
These packets should be going out eth2 not eth1 eth1 y.y.y.174 tcpdump -pnli eth1 host 131.247.254.5
16:23:06.097415 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 740341557, win 92, options [nop,nop,TS val 423546741 ecr 2565444858], length 0 16:23:06.317381 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423546961 ecr 2565445077,nop,nop,sack 1 {4294967201:1}], length 0 16:23:06.764908 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423547408 ecr 2565445515,nop,nop,sack 1 {4294967201:1}], length 0 16:23:07.634894 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423548278 ecr 2565446391,nop,nop,sack 1 {4294967201:1}], length 0 16:23:09.394972 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423550038 ecr 2565448143,nop,nop,sack 1 {4294967201:1}], length 0 16:23:12.887529 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423553531 ecr 2565451647,nop,nop,sack 1 {4294967201:1}], length 0 16:23:19.892616 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423560536 ecr 2565458655,nop,nop,sack 1 {4294967201:1}], length 0 16:23:33.907736 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423574551 ecr 2565472671,nop,nop,sack 1 {4294967201:1}], length 0 16:23:40.173991 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423580817 ecr 2565472671], length 13 16:23:40.388692 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423581032 ecr 2565472671], length 13 16:23:40.819714 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423581463 ecr 2565472671], length 13 16:23:41.680729 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423582324 ecr 2565472671], length 13 16:23:43.404732 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423584048 ecr 2565472671], length 13 16:23:46.852787 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423587496 ecr 2565472671], length 13 16:23:53.756879 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423594400 ecr 2565472671], length 13 16:24:01.935822 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423602578 ecr 2565500703,nop,nop,sack 1 {4294967201:1}], length 0 16:24:07.549037 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423608192 ecr 2565500703], length 13 16:24:35.133346 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423635776 ecr 2565500703], length 13 16:24:57.994150 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423658636 ecr 2565556767,nop,nop,sack 1 {4294967201:1}], length 0 16:25:30.365963 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423691008 ecr 2565556767], length 13 16:26:50.108488 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [.], ack 1, win 92, options [nop,nop,TS val 423770749 ecr 2565668895,nop,nop,sack 1 {4294967201:1}], length 0 16:27:20.703243 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [P.], seq 0:13, ack 1, win 92, options [nop,nop,TS val 423801344 ecr 2565668895], length 13 16:28:06.104578 IP x.x.x.217.53651 > 131.247.254.5.ftp: Flags [F.], seq 13, ack 16, win 92, options [nop,nop,TS val 423846744 ecr 2565744900], length 0
Is there a way to make this work correctly?
Thanks, Steve