[CentOS] Route traffic through private IP for only certain hosts

Hi

I am having a weird problem which I cant figure out - so I was hoping
someone here could give me a hand.

First off the end goal is that a specific server in my network runs an
IPSEC connection to another company and I want all other servers to route
traffic for the IP on that network through this single server.

Server 1 in this example is the server that runs the IPSEC connection.
(CentOS 6.6)

Server 2 in this example is an app server that would route traffic for only
that specific IP through server 1. (CentOS 6.5)

**Some IP's that will be used below:**

Server 1
<pre>
Server 1 Public IP: x.x.x.x
Server 1 Public Broadcast: x.x.x.y
Server 1 Public Gateway: x.x.x.z
Server 1 Internal IP: 10.0.64.10/24
</pre>

Server 2
<pre>
Server 2 Public IP: y.y.y.y
Server 2 Public Broadcast: y.y.y.z
Server 2 Public Gateway: y.y.y.a
Server 2 Internal IP: 10.0.64.150/24
</pre>

Those servers have full connectivity between them internally (i.e. I can
ping, ssh etc from one to the other without problem). They also both have
full acceess to the internet and can be reached that way


----------


**Server 1**

Here is an *ip a* for that

<pre># ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
    link/ether 00:0c:29:99:12:85 brd ff:ff:ff:ff:ff:ff
    inet x.x.x.x/28 brd x.x.x.y scope global eth0
    inet6 xxxx:xxxx:xxxx:xxxx/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state
UP qlen 1000
    link/ether 00:0c:29:99:12:8f brd ff:ff:ff:ff:ff:ff
    inet 10.0.64.10/24 brd 10.0.64.255 scope global eth1
    inet6 fe80::20c:29ff:fe99:128f/64 scope link
       valid_lft forever preferred_lft forever
</pre>

Here is an *ip route*
<pre># ip route
x.x.x.y/28 dev eth0  proto kernel  scope link  src x.x.x.x
10.0.64.0/24 dev eth1  proto kernel  scope link  src 10.0.64.10
169.254.0.0/16 dev eth0  scope link  metric 1002
169.254.0.0/16 dev eth1  scope link  metric 1003
default via x.x.x.z dev eth0
</pre>

Here is a *sysctl -p*
<pre>
# sysctl -p
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.default.proxy_arp = 1
net.ipv4.conf.all.rp_filter = 1
kernel.sysrq = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 1
</pre>


----------

**Server 2**

I've added a single test ip (8.8.8.8) to server two to test if it works
before bringing IPSEC into the equation

Here is an *ip a*
<pre>
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
1000
    link/ether 00:0c:29:15:8b:01 brd ff:ff:ff:ff:ff:ff
    inet y.y.y.y/29 brd y.y.y.z scope global eth0
    inet6 fe80::20c:29ff:fe15:8b01/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen
1000
    link/ether 00:0c:29:15:8b:0b brd ff:ff:ff:ff:ff:ff
    inet 10.0.64.150/24 brd 10.0.64.255 scope global eth1
    inet6 fe80::20c:29ff:fe15:8b0b/64 scope link
       valid_lft forever preferred_lft forever
</pre>

Here is an *ip route*
<pre>
# ip route
8.8.8.8 via 10.0.64.10 dev eth1
y.y.y.z/29 dev eth0  proto kernel  scope link  src y.y.y.y
10.0.64.0/24 dev eth1  proto kernel  scope link  src 10.0.64.150
default via y.y.y.a dev eth0
</pre>


----------
Now when I try do a ping from Server 2 -> 8.8.8.8 here are the tcpdumps
from each server:

**Server 2**

If I tcpdump on eth0 i get no matches (so the route appears right!). eth1
gets matches:
<pre>
# tcpdump -vvv -i eth1 -n host 8.8.8.8
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535
bytes
11:25:55.609902 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP
(1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 1, length 64
11:25:56.609262 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP
(1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 2, length 64
</pre>

**Server 1 (The hopeful gateway for 8.8.8.8)**

On eth1 (Private)
<pre>
# tcpdump -vv -i eth1 -n host 8.8.8.8
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 65535
bytes

11:27:20.608766 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP
(1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 86, length 64
11:27:21.608738 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP
(1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 87, length 64
</pre>

On eth0 (public)
<pre>
# tcpdump -vv -i eth0 -n host 8.8.8.8
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535
bytes
11:29:04.608773 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP
(1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 190, length 64
11:29:05.608800 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto ICMP
(1), length 84)
    10.0.64.150 > 8.8.8.8: ICMP echo request, id 17999, seq 191, length 64
</pre>

I've disabled the FW on both (as a test), made sure to not have any
blocking rules on FORWARD traffic (as a separate test) and I just never get
my traffic through from Server 2 to 8.8.8.8. I've also tried substituting
8.8.8.8 for another server that is reachable from both servers and the same
thing happens.

I'm open to any suggestions - i'm super confused :)

Thanks in advance,
Ian