[CentOS] CentOS-6.7, kvm bridges, virtual interfaces, and routes

I have been looking at this problem on and off for a considerable
period.  Given my lack of knowledge I have been unable to resolve this
quickly and in consequence it has been constantly shoved to the
background as other issues arise.

Here is the situation:

I have two dual-homed kvm hosts both running CentOS-6.7 and
identically configured.  These are connected to the same LAN segment
via br0/eth0 and to each other via a cross-over cable on br1/eth1. 
The IPv4 assigned to br0 on both is a publicly routeable address.  The
IPv4 assigned to br1 on both is a private address in the
192.168.0.0/16 address space.

The guests on each host have their virtual eth0 bridged with their
host's br0 and eth1 bridged with their host's br1.  The addresses used
by the guests on eth0 are publicly routeable, the addresses used on
eth1 are private.

I would like to configure br1/eth1 on both kvm hosts such that each is
a gateway to the other.

I then also would like to configure each kvm guest of each host such
that their traffic to the private network segment on the opposite host
routes through the x-over cable via br0 whilst everything else goes
out through br1 to the LAN and gateway.

Has anyone here done anything like this? If so, can you point me to
any online resource that could more or less walk me through the
process without me having to complete the coursework for a network
engineer.  I just want to keep data transfer traffic between pairs of
kvm guests off of the public lan without having to install more
hardware. The existing cabinets are not going to support it either
space wise or power wise.


An ASCII art diagram might help, or might not.

<pre>

kvmh1g1           eth0/192.168.51.1
                  eth1/aaa.bbb.ccc.151 <-------------> |
                                                       |
kvmh1             br1/aaa.bbb.ccc.51                   |
            |---> br0/192.168.51.1                     |
            X                                          |
kvmh2       |---> br0/192.168.52.1                     |
                  br1/aaa.bbb.ccc.52                   |
                                                       |
kvmh2g1           eth0/192.168.52.1                    |
                  eth1/aaa.bbb.ccc.251 <-------------> |
                                                       |
gateway           eth1/aaa.bbb.ccc.1 <---------------> |

</pre>

I have tried multiple approaches without success and in so many
variations that I no long can clearly recall the details.

At the moment my thought is that if br0 was set to 192.168.51.1/24 on
kvmh1 and to 192.168.52.1/24 on kvmh2 and a routing table entry was
made on kvmh1 to send traffic addressed to 192.168.52.0/24 through
192.168.51.1/24. And on kvmh2 br0 was set to 192.168.52.1/24 and a
routing table entry was made on kvmh2 to route traffic to
192.168.51.0/24 through 192.168.52.1/24.

I thought that if the kvm virtual guests on kvmh1 were then configured
to use addresses from 192.168.51.0/24 while those on kvmh2 used
192.168.52.0/24. And everything was configured to use their respective
host's br1 address as their gateway then this should work.  But I am
evidently either fundamentally wrong or I have misconfigured things
somehow.

Should this set up work as I imaging?  What would be the correct
static routing table entries to make it work?

-- 
***          e-Mail is NOT a SECURE channel          ***
        Do NOT transmit sensitive data via e-Mail
James B. Byrne                mailto:ByrneJB at Harte-Lyne.ca
Harte & Lyne Limited          http://www.harte-lyne.ca
9 Brockley Drive              vox: +1 905 561 1241
Hamilton, Ontario             fax: +1 905 561 0757
Canada  L8E 3C3