[CentOS] Poor perfmance of bridged interfaces

Wed Nov 25 18:24:10 UTC 2015
Gordon Messmer <gordon.messmer at gmail.com>

On 11/25/2015 07:18 AM, Sergio Belkin wrote:
> Are you talking bearing in mind VirtualBox?

Actually, no.  I'd either missed that detail or forgotten it by that 
time in the conversation.  This might be the wrong list for VirtualBox 
questions, since it's not distributed with CentOS.  I've worked with 
VirtualBox only a little bit, and I'm not sure what you're trying to do 
is possible.  I think it is, but I can only really describe how to 
accomplish what you're trying to do with respect to libvirt/KVM (the 
virtualization platform that's distributed with CentOS).

Anyway, the first thing you'd want to do is create a bridge interface 
with no slaves, and then create two VMs.  They should be attached to 
that bridge device.  (In virtualbox you might be able to do that with 
"bridged networking" to br0, or you might be able to skip the virt host 
bridge and use "internal networking".  Try the former first.)  The two 
VMs will not be attached to the real network, but should be able to 
reach each other.  Now your topology looks like this:

  ---| Network |---| (eth0)-VM Host-(br0) |--+| (eth0)-VM1 |
                                             \| (eth0)-VM2 |

...your virtualization host is connected to the real network on eth0. 
The two VMs are attached to its br0 interface.  They each have one 
"ethernet" interface and can communicate with each other.

Once you've set that up and verified that it works, set up a new bridge 
interface, br1.  Move the IP configuration from eth0 to br1, and make 
eth0 a slave of br1.  Make sure the virtualization host has network 
connectivity.  Your network topology looks exactly like it did before, 
except that the virt host uses br1 (with eth0 as a slave) to connect to 
the real network.

  ---| Network |---| (br1)-VM Host-(br0) |--+| (eth0)-VM1 |
                                            \| (eth0)-VM2 |

Next, you can add a second interface to VM1.  This one will be connected 
to br1.  (again, in virtualbox, you might be able to do this with 
bridged networking.)  Give VM1's new interface (it's eth1) an IP 
configuration appropriate for your real network, and verify that it has 
full internet connectivity.  Now your network looks like this:

                          /------------------| (eth1)\
  ---| Network |---| (br1)-VM Host-(br0) |--+| (eth0)-VM1 |
                                            \| (eth0)-VM2 |

So, now VM2 can reach VM1, but not the real network, and VM1 can reach 
the real network.

Finally, you'll create a bridge interface in VM1, and make both of its 
"ethernet" interfaces slaves.  Move its IP configuration from eth1 to 
its new br0, and drop the IP configuration from eth0.  This will give 
VM2 connectivity to the real network, so update its IP configuration to 
be compatible with the real network.  Now your network looks like this:

                         /------------------| (eth1)\
---| Network |---| (br1)-VM Host-(br0) |--+| (eth0)-(br0)-VM1 |
                                           \| (eth0)-VM2 |

When VM2 transmits a packet, it will be sent out VM2-eth0, across VM 
Host br0 to VM1-eth0, across VM1-br0 to VM Host-br1, to the real network.

And most importantly, no bridge connects two interfaces to the same 
broadcast domain.  VM Host's br0 creates a collision domain spanning the 
two VMs.  VM Host's br1 creates a collision domain spanning the physical 
network and VM1's eth1 interface.  And VM1's br0 bridges those two 
collision domains.  No loops.

I don't know what performance will look like under VirtualBox, since I 
don't use it for this sort of thing.  However, under KVM/libvirt, 
performance should be fairly good.