I have been looking at this problem on and off for a considerable period. Given my lack of knowledge I have been unable to resolve this quickly and in consequence it has been constantly shoved to the background as other issues arise.

Here is the situation:

I have two dual-homed kvm hosts both running CentOS-6.7 and identically configured. These are connected to the same LAN segment via br0/eth0 and to each other via a cross-over cable on br1/eth1. The IPv4 assigned to br0 on both is a publicly routeable address. The IPv4 assigned to br1 on both is a private address in the 192.168.0.0/16 address space.

The guests on each host have their virtual eth0 bridged with their host's br0 and eth1 bridged with their host's br1. The addresses used by the guests on eth0 are publicly routeable, the addresses used on eth1 are private.

I would like to configure br1/eth1 on both kvm hosts such that each is a gateway to the other.

I then also would like to configure each kvm guest of each host such that their traffic to the private network segment on the opposite host routes through the x-over cable via br0 whilst everything else goes out through br1 to the LAN and gateway.

Has anyone here done anything like this? If so, can you point me to any online resource that could more or less walk me through the process without me having to complete the coursework for a network engineer. I just want to keep data transfer traffic between pairs of kvm guests off of the public lan without having to install more hardware. The existing cabinets are not going to support it either space wise or power wise.

An ASCII art diagram might help, or might not.

<pre>

kvmh1g1 eth0/192.168.51.1 eth1/aaa.bbb.ccc.151 <-------------> | | kvmh1 br1/aaa.bbb.ccc.51 | |---> br0/192.168.51.1 | X | kvmh2 |---> br0/192.168.52.1 | br1/aaa.bbb.ccc.52 | | kvmh2g1 eth0/192.168.52.1 | eth1/aaa.bbb.ccc.251 <-------------> | | gateway eth1/aaa.bbb.ccc.1 <---------------> |

</pre>

I have tried multiple approaches without success and in so many variations that I no long can clearly recall the details.

At the moment my thought is that if br0 was set to 192.168.51.1/24 on kvmh1 and to 192.168.52.1/24 on kvmh2 and a routing table entry was made on kvmh1 to send traffic addressed to 192.168.52.0/24 through 192.168.51.1/24. And on kvmh2 br0 was set to 192.168.52.1/24 and a routing table entry was made on kvmh2 to route traffic to 192.168.51.0/24 through 192.168.52.1/24.

I thought that if the kvm virtual guests on kvmh1 were then configured to use addresses from 192.168.51.0/24 while those on kvmh2 used 192.168.52.0/24. And everything was configured to use their respective host's br1 address as their gateway then this should work. But I am evidently either fundamentally wrong or I have misconfigured things somehow.

Should this set up work as I imaging? What would be the correct static routing table entries to make it work?