[CentOS] Not firewall, but what?

Sun May 9 11:03:27 UTC 2010

On 05/08/2010 11:28 PM, JohnS wrote:
> If I were you I would start from scratch and go step by step and set
> it up.
> 
> John

I'm in agreement with John here.  Your set up looks complex and may be
starting from scratch is the way to go.  Looking back though the thread,
your set up might also be unnecessarily complex.  I'm not entirely sure
why you have two xen bridges and two vlans on the xen host, but that may
just be the way the xen setup scripts work, and, well, I've never used
xen. (I use libvirt with KVM, plus virtio to give me decent performance,
and I hardwire my bridges in my network-scripts so I can ensure they are
up and running before anything tries to use them).

I think the problem is either ARP or routing (or both) or an overly
complex setup (or all three) ;-)  Lets start with ARP.

ARP is pretty reliable and usually 'just works' on most networks -- so
much so that it rarely causes any problems and is easy to forget about.
The one situation that is guaranteed to cause a problem is having two
host with the same IP address on the same network. (I had a situation
once where a technician reset a problematic UPS to its defaults --
giving it the same IP address as one of our core routers and causing us
no end of problems till we figured it out).  This situation is tricky to
troubleshoot because some parts of the network may be quite stable
depending on the layout of bridges. (I had another situation once where
a wireless router was reset to defaults and took on the IP address of
the main gateway -- one bunch of Linux machines behind one switch was
stable whereas another bunch of Windows machines behind a different
switch was unstable with arp entries flip-flopping every 10 minutes or
so). I mention this case in detail because: 1) you're migrating from an
existing setup and there is a small chance that the old host is still on
the network and has the problematic IP address, or there is some static
APR entry lying around somewhere, and double checking may save you some
sanity; and 2) you're using virtualization and bridges which means ARP
has to flow and (in complex set ups) is why turning on STP can help.

For completeness (and the archives) detecting the 'multiple hosts with
the same IP' case is pretty easy: just install arpwatch on a host that
receives traffic from all hosts on the network (router, dns server, dhcp
server).  You get a nice little email every time it thinks an IP address
is flip-flopping.  You also get a nice little email every time someone
plugs a new device into your network (say, the cleaner's laptop at 2am).
If you don't have arpwatch, then you could try:

	while true; do arp -n >> arp.log; sleep 5; done

for 10-15 minutes (longer on a quiet network) and then do

	sort -u arp.log

If the same IP address pops up twice, you have a problem.  Tracking down
the culprit is a little harder, but you can use the fact that the
leading portion of the HW address is usually the same for the same NIC
manufacturer.  If the offending HW address is similar to those used by,
say, known SUN machines, then that rules out any, say, Dell desktops.
The arp.log file you just created can be handy for making these
comparisons.

The above problem is rare.  I've only experienced on 'real' hardware 3
times in the passed 10 years, but I've seen it a lot more since I
started playing with virtualization.  A more common issue is ARP
requests not flowing around your network properly.  Bridges allow you to
join networks together by passing along ARP queries.  If you have
multiple bridges, you can get loops, and this can prevent the ARP query
reaching the right bridge, so you need STP to ensure bridges cooperate
appropriately. (You can get by without STP, but only if you are 100%
sure there are no loops).  Firewalls can stop ARP flowing as well.  On
Linux this is arptables and ebtables.  I note that your xen startup
scripts turn this off, so you should be good, provided turning them off
doesn't block ARP forwarding. Not sure about that. (I notice they are on
on my systems, but I like to restrict traffic on the bridge, with the
iptables/arptable/ebtables configuration managed by shorewall).

I troubleshoot the above ARP issue the same way as for routing and
firewalls, so lets move onto that.  For newbies and the archives, I'll
detail my approach.

Firstly, you want to map out whether or not traffic can pass between
various IP addresses.  These flows may or may not be bi-directional (A
to B, is not necessarily the same as B to A) depending on stateful
firewalls and convoluted routing.  If there are more than 2 addresses
involved, things can get complicated and its easy miss important clues,
so its good to be organised.  I usually grab a scrap piece of paper and
draw up a NxN matrix of the problematic IP addresses so I can note which
flows pass and which flows fail.  I also add a few good IP addresses
(your desktop, routers, dhcp server, dns server, xen host, other guests,
home desktop, are good candidates) so I can verify my methodology and
potentially spot related issues -- these may provide clues as to the
real problem.  This may seem excessive, but at least you know you are
getting a complete picture.

I test flows with a combination of ping (or hping) and tcpdump.  I
normally layout 4 terminals in a 2x2 grid like so:

	host A: tcpdump 	host A: ping host B

	host B: tcpdump 	host B: ping host A

For testing routes and ARP my tcpdump invocation is something like

	tcpdump -i ethX icmp or arp

where ethX is the interface with the IP address I'm testing. For testing
firewall rules I use something like

	tcpdump -i ethX port YYYY

and invoke hping like so

	hping -S -p YYYY aaa.bbb.ccc.ddd

where YYYY is the port you are testing and aaa.bbb.ccc.ddd the IP
address.  There are lots of other tcpdump options that you can use to
refine what you see and cut out any irrelevant traffic (like your ssh
session).  Note that I only run one ping test at a time so I can watch
packets flowing from both sides.  The extra terminal is just to make it
a little easier to be systematic in tracing reverse flows.

Now if you do the above tests on a pair of IP addresses where routing
and ARP is good, you should see nice little ICMP request/reply pairs and
nice little ARP who-has/reply pairs. If APR isn't working, you'll be
getting a strings of ARP who-has packets with no replies.  If routing is
broken, you get strings of ICMP requests without replies (yeah
obviously) and by comparing the outputs in the three terminals, you may
be able to figure out where its broken. If you fill out your flow
matrix, you'll have a pretty clear picture of the situation and may have
some more clues as to were the problem lies.  You'll also have a nice
framework for testing potential fixes.

Jussi, if this is all old news to you, please forgive my rambling
diatribe. From bitter experience, I just like to make sure everyone's on
the same page with this stuff:-)

Now lets look at some details.

> Physical if cards:
> Intel Corporation 82567LM-3 Gigabit Network Connection
> (eth0; on motherboard)

Recently I've had some onboard NICs slowly die over a period of about 2
weeks.  When it happens you can get some fairly inexplicable errors.
Probably not your problem, but thought I might point out the possibility.

ip route output on the xen host:

> 62.236.221.64/28 dev eth0  proto kernel  scope link src 62.236.221.67
> 62.220.237.96/27 dev eth1  proto kernel  scope link src 62.220.237.104
> default via 62.220.237.126 dev eth1

Your 'good' interface eth1 is also the default route.  I gather
62.220.237.126 is your border router.  Where does your 'bad' interface
eth0 go then?  The most common problem with routing on multi-homed
systems is having packets leaving one interface and responses coming
back on another.

Really need to see what 'ip route' says on each of you guests.

Hmmm... ifconfig on you xen host:

> vif5.0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
> ....
> vif5.1    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
> ...
> xenbr0    Link encap:Ethernet  HWaddr FE:FF:FF:FF:FF:FF
>           UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1

Normally an interface with a 'dot' suffix is on a specific VLAN.  So
you've got two VLANs running here?  This complicated things.  Also your
bridge has NOARP set which does not make much sense to me.

brctl show
> bridge name     bridge id               STP enabled     interfaces
> virbr0          8000.000000000000       yes
> xenbr0          8000.feffffffffff       no              vif5.0
>                                                         vif4.0
>                                                         peth0
>                                                         vif0.0
> xenbr1          8000.feffffffffff       no              vif5.1
>                                                         vif3.0
>                                                         vif2.0
>                                                         peth1
>                                                         vif0.1

Okay, that makes my head hurt.  Why two VLANs?  What's you mapping
between virtual interfaces and guests? And which guest is the bad one?
Hmmm ... I could interpret this as meaning vif5.0 and vif5.1 are two
interfaces on guest5 which you want to directly tie to to different
physical cards.  Perhaps some source based routing on your xen host
would serve you better.  This may be part of the overly complex setup
which is causing problems.

Are these guests all just web servers?

Sorry if that is a too long ramble.  I'm a bit confused by your set up,
and as John suggested, you may need to rebuild things from scratch to
sort things out.

Hope this helps!

Kal