A CentOS 6.3 box ("host") runs several KVM virtual machines, each of which has two interfaces attached to the two bridges br1 and br2 (and each thus has two IP's; one on 192.168.0.0/22 and one on 192.168.4.0/22); net.ipv4.ip_forward on the host is 1. Simplified diagram:
host +---------------+ | | net1 = 192.168.0.0/22 | | net2 = 192.168.4.0/22 -----------------------+ br1 br2 +--------------------------------- | | | | | | | | Client A +---------------+ Client B (hosts KVM1, KVM2, etc)
Each client uses the bridge's IP address on the same side as default gateway. Client A can successfully ping or ssh (for example) to a KVM machine by IP address by using the KVM machine's net1 IP address. Client B can likewise communicate using the KVM machine's net2 IP address. However, neither client can communicate by using the address on the opposing segment (eg, Client A using KVM1_net2_IP); I can see from tcpdump that the packets are received by the virtual machine but no reply is ever made. Any clue?
Steve
On Mon, Sep 10, 2012 at 11:34 AM, Steve Thompson smt@vgersoft.com wrote:
A CentOS 6.3 box ("host") runs several KVM virtual machines, each of which has two interfaces attached to the two bridges br1 and br2 (and each thus has two IP's; one on 192.168.0.0/22 and one on 192.168.4.0/22); net.ipv4.ip_forward on the host is 1. Simplified diagram:
host +---------------+ | |net1 = 192.168.0.0/22 | | net2 = 192.168.4.0/22 -----------------------+ br1 br2 +--------------------------------- | | | | | | | | Client A +---------------+ Client B (hosts KVM1, KVM2, etc)
Each client uses the bridge's IP address on the same side as default gateway. Client A can successfully ping or ssh (for example) to a KVM machine by IP address by using the KVM machine's net1 IP address. Client B can likewise communicate using the KVM machine's net2 IP address. However, neither client can communicate by using the address on the opposing segment (eg, Client A using KVM1_net2_IP); I can see from tcpdump that the packets are received by the virtual machine but no reply is ever made. Any clue?
Routing problem? What are the response to each of the commands below on all five systems: host, Client A and B, KVM1 and 2:
# ip addr show # ip route show
On Mon, 10 Sep 2012, Dale Dellutri wrote:
Routing problem?
Not that I can see, but here is the info (omitting interfaces that are not up). I included on one KVM since the problem is common to the others, and they are all set up the same way.
On the host:
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 84:2b:2b:47:e8:7d brd ff:ff:ff:ff:ff:ff inet6 fe80::862b:2bff:fe47:e87d/64 scope link valid_lft forever preferred_lft forever 4: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:6f:2b:4c brd ff:ff:ff:ff:ff:ff inet6 fe80::21b:21ff:fe6f:2b4c/64 scope link valid_lft forever preferred_lft forever 7: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 84:2b:2b:47:e8:7d brd ff:ff:ff:ff:ff:ff inet 192.168.4.2/22 brd 192.168.7.255 scope global br1 inet6 fe80::862b:2bff:fe47:e87d/64 scope link valid_lft forever preferred_lft forever 8: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 00:1b:21:6f:2b:4c brd ff:ff:ff:ff:ff:ff inet 192.168.0.2/22 brd 192.168.3.255 scope global br2 inet6 fe80::21b:21ff:fe6f:2b4c/64 scope link valid_lft forever preferred_lft forever 10: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 52:54:00:9d:ad:f7 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 11: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500 link/ether 52:54:00:9d:ad:f7 brd ff:ff:ff:ff:ff:ff 12: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether fe:54:00:13:73:28 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe13:7328/64 scope link valid_lft forever preferred_lft forever 13: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether fe:54:00:12:99:dd brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe12:99dd/64 scope link valid_lft forever preferred_lft forever 14: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether fe:54:00:72:f5:33 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe72:f533/64 scope link valid_lft forever preferred_lft forever
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 192.168.4.0/22 dev br1 proto kernel scope link src 192.168.4.2 192.168.0.0/22 dev br2 proto kernel scope link src 192.168.0.2 default via 192.168.0.1 dev br2
On KVM1:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:13:73:28 brd ff:ff:ff:ff:ff:ff inet 192.168.3.253/22 brd 192.168.3.255 scope global eth0 inet6 fe80::5054:ff:fe13:7328/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:12:99:dd brd ff:ff:ff:ff:ff:ff inet 192.168.7.253/22 brd 192.168.7.255 scope global eth1 inet6 fe80::5054:ff:fe12:99dd/64 scope link valid_lft forever preferred_lft forever
192.168.4.0/22 dev eth1 proto kernel scope link src 192.168.7.253 192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.3.253 default via 192.168.0.1 dev eth0
On client A:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:14:22:27:9b:51 brd ff:ff:ff:ff:ff:ff inet 192.168.0.172/22 brd 192.168.3.255 scope global eth0 inet6 fe80::214:22ff:fe27:9b51/64 scope link valid_lft forever preferred_lft forever
192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.0.172 default via 192.168.0.2 dev eth0
On client B:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:19:b9:c7:23:ad brd ff:ff:ff:ff:ff:ff inet 192.168.5.241/22 brd 192.168.7.255 scope global eth0 inet6 fe80::219:b9ff:fec7:23ad/64 scope link valid_lft forever preferred_lft forever
192.168.4.0/22 dev eth0 proto kernel scope link src 192.168.5.241 default via 192.168.4.1 dev eth0
-Steve
On Mon, Sep 10, 2012 at 12:37 PM, Steve Thompson smt@vgersoft.com wrote:
On Mon, 10 Sep 2012, Dale Dellutri wrote:
Routing problem?
Not that I can see, but here is the info (omitting interfaces that are not up). I included on one KVM since the problem is common to the others, and they are all set up the same way.
On the host:
3: em2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 84:2b:2b:47:e8:7d brd ff:ff:ff:ff:ff:ff inet6 fe80::862b:2bff:fe47:e87d/64 scope link valid_lft forever preferred_lft forever 4: p1p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 00:1b:21:6f:2b:4c brd ff:ff:ff:ff:ff:ff inet6 fe80::21b:21ff:fe6f:2b4c/64 scope link valid_lft forever preferred_lft forever 7: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 84:2b:2b:47:e8:7d brd ff:ff:ff:ff:ff:ff inet 192.168.4.2/22 brd 192.168.7.255 scope global br1 inet6 fe80::862b:2bff:fe47:e87d/64 scope link valid_lft forever preferred_lft forever 8: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 00:1b:21:6f:2b:4c brd ff:ff:ff:ff:ff:ff inet 192.168.0.2/22 brd 192.168.3.255 scope global br2 inet6 fe80::21b:21ff:fe6f:2b4c/64 scope link valid_lft forever preferred_lft forever 10: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN link/ether 52:54:00:9d:ad:f7 brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 11: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 500 link/ether 52:54:00:9d:ad:f7 brd ff:ff:ff:ff:ff:ff 12: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether fe:54:00:13:73:28 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe13:7328/64 scope link valid_lft forever preferred_lft forever 13: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether fe:54:00:12:99:dd brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe12:99dd/64 scope link valid_lft forever preferred_lft forever 14: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500 link/ether fe:54:00:72:f5:33 brd ff:ff:ff:ff:ff:ff inet6 fe80::fc54:ff:fe72:f533/64 scope link valid_lft forever preferred_lft forever
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 192.168.4.0/22 dev br1 proto kernel scope link src 192.168.4.2 192.168.0.0/22 dev br2 proto kernel scope link src 192.168.0.2 default via 192.168.0.1 dev br2
On KVM1:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:13:73:28 brd ff:ff:ff:ff:ff:ff inet 192.168.3.253/22 brd 192.168.3.255 scope global eth0 inet6 fe80::5054:ff:fe13:7328/64 scope link valid_lft forever preferred_lft forever 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 52:54:00:12:99:dd brd ff:ff:ff:ff:ff:ff inet 192.168.7.253/22 brd 192.168.7.255 scope global eth1 inet6 fe80::5054:ff:fe12:99dd/64 scope link valid_lft forever preferred_lft forever
192.168.4.0/22 dev eth1 proto kernel scope link src 192.168.7.253 192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.3.253 default via 192.168.0.1 dev eth0
On client A:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:14:22:27:9b:51 brd ff:ff:ff:ff:ff:ff inet 192.168.0.172/22 brd 192.168.3.255 scope global eth0 inet6 fe80::214:22ff:fe27:9b51/64 scope link valid_lft forever preferred_lft forever
192.168.0.0/22 dev eth0 proto kernel scope link src 192.168.0.172 default via 192.168.0.2 dev eth0
On client B:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:19:b9:c7:23:ad brd ff:ff:ff:ff:ff:ff inet 192.168.5.241/22 brd 192.168.7.255 scope global eth0 inet6 fe80::219:b9ff:fec7:23ad/64 scope link valid_lft forever preferred_lft forever
192.168.4.0/22 dev eth0 proto kernel scope link src 192.168.5.241 default via 192.168.4.1 dev eth0
This looks like it should work for Client A, but maybe not for Client B (see below). So maybe it's a firewall problem (iptables chain FORWARD) on the host?
Client B's default route is 192.168.4.1. This address is not on the host. Did you mean to use .2? If not, is .1 aware of the routing to the 192.168.0.0/22 network?
On Mon, 10 Sep 2012, Dale Dellutri wrote:
This looks like it should work for Client A, but maybe not for Client B (see below). So maybe it's a firewall problem (iptables chain FORWARD) on the host?
Client B's default route is 192.168.4.1. This address is not on the host. Did you mean to use .2? If not, is .1 aware of the routing to the 192.168.0.0/22 network?
Actually I have two similar setups, one with .1 and one with .2, so I mixed up the examples here. But in reality it is setup up correctly. And it doesn't work for either client :-(
Steve
On Mon, 10 Sep 2012, Steve Thompson wrote:
On Mon, 10 Sep 2012, Dale Dellutri wrote:
This looks like it should work for Client A, but maybe not for Client B (see below). So maybe it's a firewall problem (iptables chain FORWARD) on the host?
Client B's default route is 192.168.4.1. This address is not on the host. Did you mean to use .2? If not, is .1 aware of the routing to the 192.168.0.0/22 network?
Actually I have two similar setups, one with .1 and one with .2, so I mixed up the examples here. But in reality it is setup up correctly. And it doesn't work for either client :-(
Let me expand on this. There is no issue with a client on net1 communicating with a client on net2; the host passes packets from one subnet to the other as it should. The only issue is when the client is a virtual machine on the host.
Steve
On Mon, 10 Sep 2012, Steve Thompson wrote:
On Mon, 10 Sep 2012, Steve Thompson wrote:
On Mon, 10 Sep 2012, Dale Dellutri wrote:
This looks like it should work for Client A, but maybe not for Client B (see below). So maybe it's a firewall problem (iptables chain FORWARD) on the host?
Let me expand on this. There is no issue with a client on net1 communicating with a client on net2; the host passes packets from one subnet to the other as it should. The only issue is when the client is a virtual machine on the host.
For those following along at home, the solution to this turned out to be related to the change in the function of the net.ipv4.conf.default.rp_filter parameter in the CentOS 6 kernels; it had nothing to do with KVM. Changing the value of rp_filter from 1 to 2 resolved all issues.
Steve
On Mon, Sep 10, 2012 at 11:34 AM, Steve Thompson smt@vgersoft.com wrote:
Each client uses the bridge's IP address on the same side as default gateway.
What does that mean? A bridge shouldn't have an address and a gateway needs to be the IP of something capable of routing.
On Mon, 10 Sep 2012, Les Mikesell wrote:
What does that mean? A bridge shouldn't have an address and a gateway needs to be the IP of something capable of routing.
Sure it has an address:
# ip addr show br1 7: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN ... inet 192.168.4.2/22 brd 192.168.7.255 scope global br1 inet6 fe80::862b:2bff:fe47:e87d/64 scope link valid_lft forever preferred_lft forever
-s
On Mon, Sep 10, 2012 at 4:45 PM, Steve Thompson smt@vgersoft.com wrote:
What does that mean? A bridge shouldn't have an address and a gateway needs to be the IP of something capable of routing.
Sure it has an address:
# ip addr show br1 7: br1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN ... inet 192.168.4.2/22 brd 192.168.7.255 scope global br1 inet6 fe80::862b:2bff:fe47:e87d/64 scope link valid_lft forever preferred_lft forever
-s
Do the things you are trying to reach have a route back through the KVM host?
-
Dale Dellutri -
Les Mikesell -
Steve Thompson