Hi
I am having a weird problem which I cant figure out - so I was hoping someone here could give me a hand.
First off the end goal is that a specific server in my network runs an IPSEC connection to another company and I want all other servers to route traffic for the IP on that network through this single server.
Server 1 in this example is the server that runs the IPSEC connection. (CentOS 6.6)
Server 2 in this example is an app server that would route traffic for only that specific IP through server 1. (CentOS 6.5)
**Some IP's that will be used below:**
Server 1 <pre> Server 1 Public IP: x.x.x.x Server 1 Public Broadcast: x.x.x.y Server 1 Public Gateway: x.x.x.z Server 1 Internal IP: 10.0.64.10/24 </pre>
Server 2 <pre> Server 2 Public IP: y.y.y.y Server 2 Public Broadcast: y.y.y.z Server 2 Public Gateway: y.y.y.a Server 2 Internal IP: 10.0.64.150/24 </pre>
Those servers have full connectivity between them internally (i.e. I can ping, ssh etc from one to the other without problem). They also both have full acceess to the internet and can be reached that way
----------
**Server 1**
Here is an *ip a* for that
<pre># ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:99:12:85 brd ff:ff:ff:ff:ff:ff inet x.x.x.x/28 brd x.x.x.y scope global eth0 inet6 xxxx:xxxx:xxxx:xxxx/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:99:12:8f brd ff:ff:ff:ff:ff:ff inet 10.0.64.10/24 brd 10.0.64.255 scope global eth1 inet6 fe80::20c:29ff:fe99:128f/64 scope link valid_lft forever preferred_lft forever </pre>
Here is an *ip route* <pre># ip route x.x.x.y/28 dev eth0 proto kernel scope link src x.x.x.x 10.0.64.0/24 dev eth1 proto kernel scope link src 10.0.64.10 169.254.0.0/16 dev eth0 scope link metric 1002 169.254.0.0/16 dev eth1 scope link metric 1003 default via x.x.x.z dev eth0 </pre>
Here is a *sysctl -p* <pre> # sysctl -p net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 kernel.sysrq = 0 kernel.core_uses_pid = 1 net.ipv4.tcp_syncookies = 1 kernel.msgmnb = 65536 kernel.msgmax = 65536 kernel.shmmax = 68719476736 kernel.shmall = 4294967296 net.ipv4.ip_forward = 1 net.ipv6.conf.all.forwarding = 1 net.ipv4.conf.default.proxy_arp = 1 net.ipv4.conf.all.rp_filter = 1 kernel.sysrq = 1 net.ipv4.conf.default.send_redirects = 1 net.ipv4.conf.all.send_redirects = 1 </pre>
----------
**Server 2**
I've added a single test ip (8.8.8.8) to server two to test if it works before bringing IPSEC into the equation
Here is an *ip a* <pre> # ip a 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:0c:29:15:8b:01 brd ff:ff:ff:ff:ff:ff inet y.y.y.y/29 brd y.y.y.z scope global eth0 inet6 fe80::20c:29ff:fe15:8b01/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:0c:29:15:8b:0b brd ff:ff:ff:ff:ff:ff inet 10.0.64.150/24 brd 10.0.64.255 scope global eth1 inet6 fe80::20c:29ff:fe15:8b0b/64 scope link valid_lft forever preferred_lft forever </pre>
Here is an *ip route* <pre> # ip route 8.8.8.8 via 10.0.64.10 dev eth1 y.y.y.z/29 dev eth0 proto kernel scope link src y.y.y.y 10.0.64.0/24 dev eth1 proto kernel scope link src 10.0.64.150 default via y.y.y.a dev eth0 </pre>
---------- Now when I try do a ping from Server 2 -> 8.8.8.8 here are the tcpdumps from each server:
**Server 2**
If I tcpdump on eth0 i get no matches (so the route appears right!). eth1 gets matches: <pre> # tcpdump -vvv -i eth1 -n host 8.8.8.8 tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 11:25:55.609902 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 1, length 64 11:25:56.609262 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 2, length 64 </pre>
**Server 1 (The hopeful gateway for 8.8.8.8)**
On eth1 (Private) <pre> # tcpdump -vv -i eth1 -n host 8.8.8.8 tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes
11:27:20.608766 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 86, length 64 11:27:21.608738 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 87, length 64 </pre>
On eth0 (public) <pre> # tcpdump -vv -i eth0 -n host 8.8.8.8 tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 11:29:04.608773 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 190, length 64 11:29:05.608800 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 191, length 64 </pre>
I've disabled the FW on both (as a test), made sure to not have any blocking rules on FORWARD traffic (as a separate test) and I just never get my traffic through from Server 2 to 8.8.8.8. I've also tried substituting 8.8.8.8 for another server that is reachable from both servers and the same thing happens.
I'm open to any suggestions - i'm super confused :)
Thanks in advance, Ian