I built a CentOS 6 machine to host several CentOS 6 guest servers. As all guests will be Internet facing I set up the host with two bridged NICs and assigned an Internet facing IP address to br0 and a local IP address to br1.
Each guest was installed using br0 and br1 with virtio drivers. On each I assigned an Internet facing IP address to eth0 and a local IP address on eth1. So far so good. I can access the guest servers from either IP address as expected. That is HTTP, SSH and SMTP servers on them are accessible and do what they are supposed to do.
Except... Except from any location outside of my Comcast Cable Modem. To be clear, from any machine inside the modem to any address on the guests, all works perfectly. But outside the modem the guest apps either don't receive packets or for some reason don't respond, and I've tired it from four different locations.
Using Wireshark on the guests I can see the packets arrive from the outside sources, but no response is seen. On accesses from inside I can see both incoming and outgoing packets, as expected.
I can ping the outside sources from the guests, yet pings from the outside sources get no response from the guests. All the outside sources get responses when pinging the host. I can ping the guests from any inside machine.
I've tried it with and without firewalls, both on the guests and the host. Including with the following iptables rule:
iptables -A FORWARD -m physdev --physdev-is-bridged -j ACCEPT
and the following sysctl adjustments:
net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0
Both of the guest servers I've built have have same issue.
Any ideas? I am at loss as to how to proceed and am about to reconsider the idea of multiple guests as servers.
All packages are from the CentOS repo and I am using kvm-qemu via libvirtd and virt-manager.
Emmett
I built a CentOS 6 machine to host several CentOS 6 guest servers. As all
guests will be Internet facing I set up the host with two bridged NICs and assigned an Internet facing IP address to br0 and a local IP address to br1.
Each guest was installed using br0 and br1 with virtio drivers. On each I
assigned an Internet facing IP address to eth0 and a local IP address on eth1. So far so good. I can access the guest servers from either IP address as expected. That is HTTP, SSH and SMTP servers on them are accessible and do what they are supposed to do.
Except... Except from any location outside of my Comcast Cable Modem. To
be clear, from any machine inside the modem to any address on the guests, all works perfectly. But outside the modem the guest apps either don't receive packets or for some reason don't respond, and I've tired it from four different locations.
Using Wireshark on the guests I can see the packets arrive from the
outside sources, but no response is seen. On accesses from inside I can see both incoming and outgoing packets, as expected.
I can ping the outside sources from the guests, yet pings from the outside
sources get no response from the guests. All the outside sources get responses when pinging the host. I can ping the guests from any inside machine.
Initial thought is a routing issue particularly with multiple NICs.
What does 'ip r s' reveal?
On 19/07/2011 08:14, James Hogarth wrote:
I built a CentOS 6 machine to host several CentOS 6 guest servers.
As all guests will be Internet facing I set up the host with two bridged NICs and assigned an Internet facing IP address to br0 and a local IP address to br1.
Each guest was installed using br0 and br1 with virtio drivers. On
each I assigned an Internet facing IP address to eth0 and a local IP address on eth1. So far so good. I can access the guest servers from either IP address as expected. That is HTTP, SSH and SMTP servers on them are accessible and do what they are supposed to do.
Except... Except from any location outside of my Comcast Cable
Modem. To be clear, from any machine inside the modem to any address on the guests, all works perfectly. But outside the modem the guest apps either don't receive packets or for some reason don't respond, and I've tired it from four different locations.
Using Wireshark on the guests I can see the packets arrive from the
outside sources, but no response is seen. On accesses from inside I can see both incoming and outgoing packets, as expected.
I can ping the outside sources from the guests, yet pings from the
outside sources get no response from the guests. All the outside sources get responses when pinging the host. I can ping the guests from any inside machine.
A bit of a long shot but does turning on STP on the br* interfaces help? I vaguely remember I had to do the following on one of my machines that uses bonding + bridges:
# brctl stp br0 on
I have put this in the machines' /etc/rc.local so it's applied upon every reboot.
On 07/19/2011 07:52 PM, Khusro Jaleel wrote:
A bit of a long shot but does turning on STP on the br* interfaces help? I vaguely remember I had to do the following on one of my machines that uses bonding + bridges:
# brctl stp br0 on
I have put this in the machines' /etc/rc.local so it's applied upon every reboot.
I've had to do the same for my Fedora hosts with Centos guests. You can put STP=on in /etc/sysconfig/network-scripts/ifcfg-br0
to force this bridge to always use STP.
Also, adding
NETWORKDELAY=15 to /etc/sysconfig/network
will force you network startup script to hang for 15 seconds during start up. This gives the STP a chance to settle before any other services attempt to use the bridge.
Kal
On 07/19/2011 12:14 AM, James Hogarth wrote:
I built a CentOS 6 machine to host several CentOS 6 guest servers. As all guests will be Internet facing I set up the host with two bridged NICs and assigned an Internet facing IP address to br0 and a local IP address to br1.
Each guest was installed using br0 and br1 with virtio drivers. On each I assigned an Internet facing IP address to eth0 and a local IP address on eth1. So far so good. I can access the guest servers from either IP address as expected. That is HTTP, SSH and SMTP servers on them are accessible and do what they are supposed to do.
Except... Except from any location outside of my Comcast Cable Modem. To be clear, from any machine inside the modem to any address on the guests, all works perfectly. But outside the modem the guest apps either don't receive packets or for some reason don't respond, and I've tired it from four different locations.
Using Wireshark on the guests I can see the packets arrive from the outside sources, but no response is seen. On accesses from inside I can see both incoming and outgoing packets, as expected.
I can ping the outside sources from the guests, yet pings from the outside sources get no response from the guests. All the outside sources get responses when pinging the host. I can ping the guests from any inside machine.
Initial thought is a routing issue particularly with multiple NICs.
What does 'ip r s' reveal?
That was it! ip r s showed that I had the local facing NIC (eth1) as the gateway, which caused all outgoing packets to be routed to the local network DUH!.
Emmett
Initial thought is a routing issue particularly with multiple NICs.
What does 'ip r s' reveal?
That was it! ip r s showed that I had the local facing NIC (eth1) as the gateway, which caused all outgoing packets to be routed to the local network DUH!.
Yup been there before.....
So long as the local NIC doesn't need to hit any other internal subnets you'll be fine... otherwise don't forget to add the appropriate RFC1918 static routes to go out that interface with the default being the external one....
Also another thing to watch out for is asymmetric routing. If it is possible to get to a destination via either interface and due to routing issues elsewhere the packet does not return to the same interface through the same gateway you can get odd behaviour.... In which case you would need to set up policy routing rules to ensure stuff enters/leaves the right way... but dont' worry about that for now - just remember it for later if you get odd network behaviour....
On 07/20/2011 01:24 AM, James Hogarth wrote:
Initial thought is a routing issue particularly with multiple NICs.
What does 'ip r s' reveal?
That was it! ip r s showed that I had the local facing NIC (eth1) as the gateway, which caused all outgoing packets to be routed to the local network DUH!.
Yup been there before.....
So long as the local NIC doesn't need to hit any other internal subnets you'll be fine... otherwise don't forget to add the appropriate RFC1918 static routes to go out that interface with the default being the external one....
Also another thing to watch out for is asymmetric routing. If it is possible to get to a destination via either interface and due to routing issues elsewhere the packet does not return to the same interface through the same gateway you can get odd behaviour.... In which case you would need to set up policy routing rules to ensure stuff enters/leaves the right way... but dont' worry about that for now - just remember it for later if you get odd network behaviour....
Thanks James,
I keep it in mind, but by the time it happens I'll need Google again to find this thread :-)
-
Emmett Culley -
James Hogarth -
Kahlil Hodgson -
Khusro Jaleel