[CentOS-virt] Arp Flip Flops make machine inaccessible.

Sat Jul 31 22:50:30 UTC 2010
Ben M. <centos at rivint.com>

CentOS 5.5 Xen "standard" Xen Installation.

I have two nics. I just put the second one to DHCP and modified the 
ifcfg-et01 and so far I am holding, but I am not confident. Prior they 
were sequential IP Addrs on same subnet.

arpwatch has indicated flip flips. I can find no rhyme or reason to 
predict them. I know I missed I must have missed a step somewhere.

I want to keep the traffic pinned to the physical ethXs by their Xen 
bridge assignation. Segregate traffic.

Current working setup:
# ifcfg-eth0
# nVidia Corporation MCP55 Ethernet
DEVICE=eth0
BOOTPROTO=none
#BROADCAST=10.255.255.255
#HWADDR=00:e0:81:d2:0b:36
IPADDR=10.0.0.50
#NETMASK=255.255.255.0
#NETWORK=10.0.0.0
ONBOOT=yes
GATEWAY=10.0.0.1
TYPE=Ethernet
#ARP=yes
IPV6INIT=no

#ifcfg-eth1
# nVidia Corporation MCP55 Ethernet
DEVICE=eth1
BOOTPROTO=dhcp
ONBOOT=yes
HWADDR=00:e0:81:d2:0b:37


[root at localdomain ~]# ifconfig -a
eth0      Link encap:Ethernet  HWaddr 00:E0:81:D2:0B:36
           inet addr:10.0.0.50  Bcast:10.255.255.255  Mask:255.0.0.0
           inet6 addr: fe80::2e0:81ff:fed2:b36/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:17853 errors:0 dropped:0 overruns:0 frame:0
           TX packets:100 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:1249977 (1.1 MiB)  TX bytes:8043 (7.8 KiB)

eth1      Link encap:Ethernet  HWaddr 00:E0:81:D2:0B:37
           inet addr:10.255.252.22  Bcast:10.255.255.255  Mask:255.0.0.0
           inet6 addr: fe80::2e0:81ff:fed2:b37/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:8631 errors:0 dropped:0 overruns:0 frame:0
           TX packets:13361 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:657672 (642.2 KiB)  TX bytes:6476669 (6.1 MiB)

lo        Link encap:Local Loopback
           inet addr:127.0.0.1  Mask:255.0.0.0
           inet6 addr: ::1/128 Scope:Host
           UP LOOPBACK RUNNING  MTU:16436  Metric:1
           RX packets:18946 errors:0 dropped:0 overruns:0 frame:0
           TX packets:18946 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:7802726 (7.4 MiB)  TX bytes:7802726 (7.4 MiB)

peth0     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:17887 errors:0 dropped:0 overruns:0 frame:0
           TX packets:137 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:1253769 (1.1 MiB)  TX bytes:11507 (11.2 KiB)
           Interrupt:252 Base address:0xa000

peth1     Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:363575 errors:0 dropped:0 overruns:0 frame:0
           TX packets:161997 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:1000
           RX bytes:500789617 (477.5 MiB)  TX bytes:17282884 (16.4 MiB)
           Interrupt:251 Base address:0xc000

sit0      Link encap:IPv6-in-IPv4
           NOARP  MTU:1480  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

tap0      Link encap:Ethernet  HWaddr 76:A2:E4:74:CD:79
           inet6 addr: fe80::74a2:e4ff:fe74:cd79/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:2652 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:500
           RX bytes:0 (0.0 b)  TX bytes:449235 (438.7 KiB)

tap1      Link encap:Ethernet  HWaddr AA:BB:F6:55:59:5E
           inet6 addr: fe80::a8bb:f6ff:fe55:595e/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:1 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8862 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:500
           RX bytes:60 (60.0 b)  TX bytes:878033 (857.4 KiB)

veth2     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

veth3     Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif0.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:100 errors:0 dropped:0 overruns:0 frame:0
           TX packets:17853 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:8043 (7.8 KiB)  TX bytes:1249977 (1.1 MiB)

vif0.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:13361 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8631 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:6476669 (6.1 MiB)  TX bytes:657672 (642.2 KiB)

vif0.2    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif0.3    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           BROADCAST MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

vif1.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:63 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8730 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:32
           RX bytes:6847 (6.6 KiB)  TX bytes:866585 (846.2 KiB)

vif3.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:179 errors:0 dropped:0 overruns:0 frame:0
           TX packets:8627 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:32
           RX bytes:15806 (15.4 KiB)  TX bytes:857989 (837.8 KiB)

vif4.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:148068 errors:0 dropped:0 overruns:0 frame:0
           TX packets:357556 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:32
           RX bytes:6980701 (6.6 MiB)  TX bytes:500418149 (477.2 MiB)

virbr0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           inet addr:192.168.122.1  Bcast:192.168.122.255 
Mask:255.255.255.0
           inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)

virbr1    Link encap:Ethernet  HWaddr 00:00:00:00:00:00
           inet addr:192.168.100.1  Bcast:192.168.100.255 
Mask:255.255.255.0
           inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
           TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:0 (0.0 b)  TX bytes:468 (468.0 b)

xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:1506 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:163593 (159.7 KiB)  TX bytes:0 (0.0 b)

xenbr1    Link encap:Ethernet  HWaddr 76:A2:E4:74:CD:79
           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
           RX packets:1503 errors:0 dropped:0 overruns:0 frame:0
           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
           collisions:0 txqueuelen:0
           RX bytes:161739 (157.9 KiB)  TX bytes:0 (0.0 b)


[root at river50 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.000000000000       yes
virbr1          8000.000000000000       yes
xenbr0          8000.feffffffffff       no              peth0
                                                         vif0.0
xenbr1          8000.76a2e474cd79       no              vif4.0
                                                         tap0
                                                         vif3.0
                                                         tap1
                                                         vif1.0
                                                         peth1
                                                         vif0.1



REPLACED in xend-config.sxp
# (network-script network-bridge)
(network-script my-network-bridge)

file: my-network-bridge
[root at localdomain ~]# cat /etc/xen/scripts/my-network-bridge
#!/bin/sh
dir=$(dirname "$0")
"$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge=xenbr0
"$dir/network-bridge" "$@" vifnum=1 netdev=eth1 bridge=xenbr1


IPTABLES (not modified by any action by me)
[root at localdomain ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:bootps
ACCEPT     udp  --  anywhere             anywhere            udp dpt:domain
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:domain
ACCEPT     udp  --  anywhere             anywhere            udp dpt:bootps
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:bootps
RH-Firewall-1-INPUT  all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  anywhere             private.ip-address.localhost/24 
state RELATED,ESTABLISHED
ACCEPT     all  --  private.ip-address.localhost/24  anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere            reject-with 
icmp-port-unreachable
REJECT     all  --  anywhere             anywhere            reject-with 
icmp-port-unreachable
ACCEPT     all  --  anywhere             private.ip-address.localhost/24 
state RELATED,ESTABLISHED
ACCEPT     all  --  private.ip-address.localhost/24  anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere            reject-with 
icmp-port-unreachable
REJECT     all  --  anywhere             anywhere            reject-with 
icmp-port-unreachable
RH-Firewall-1-INPUT  all  --  anywhere             anywhere
ACCEPT     all  --  private.ip-address.localhost  anywhere 
PHYSDEV match --physdev-in vif1.0
ACCEPT     udp  --  anywhere             anywhere            PHYSDEV 
match --physdev-in vif1.0 udp spt:bootpc dpt:bootps
ACCEPT     all  --  anywhere             anywhere            PHYSDEV 
match --physdev-in vif3.0
ACCEPT     all  --  private.ip-address.localhost  anywhere 
PHYSDEV match --physdev-in vif4.0
ACCEPT     udp  --  anywhere             anywhere            PHYSDEV 
match --physdev-in vif4.0 udp spt:bootpc dpt:bootps

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain RH-Firewall-1-INPUT (2 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere            icmp any
ACCEPT     esp  --  anywhere             anywhere
ACCEPT     ah   --  anywhere             anywhere
ACCEPT     udp  --  anywhere             224.0.0.251         udp dpt:mdns
ACCEPT     udp  --  anywhere             anywhere            udp dpt:ipp
ACCEPT     tcp  --  anywhere             anywhere            tcp dpt:ipp
ACCEPT     all  --  anywhere             anywhere            state 
RELATED,ESTABLISHED
ACCEPT     tcp  --  anywhere             anywhere            state NEW 
tcp dpt:5901
ACCEPT     tcp  --  anywhere             anywhere            state NEW 
tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            state NEW 
tcp dpt:smtp
REJECT     all  --  anywhere             anywhere            reject-with 
icmp-host-prohibited