On 05/08/2010 11:28 PM, JohnS wrote: > If I were you I would start from scratch and go step by step and set > it up. > > John I'm in agreement with John here. Your set up looks complex and may be starting from scratch is the way to go. Looking back though the thread, your set up might also be unnecessarily complex. I'm not entirely sure why you have two xen bridges and two vlans on the xen host, but that may just be the way the xen setup scripts work, and, well, I've never used xen. (I use libvirt with KVM, plus virtio to give me decent performance, and I hardwire my bridges in my network-scripts so I can ensure they are up and running before anything tries to use them). I think the problem is either ARP or routing (or both) or an overly complex setup (or all three) ;-) Lets start with ARP. ARP is pretty reliable and usually 'just works' on most networks -- so much so that it rarely causes any problems and is easy to forget about. The one situation that is guaranteed to cause a problem is having two host with the same IP address on the same network. (I had a situation once where a technician reset a problematic UPS to its defaults -- giving it the same IP address as one of our core routers and causing us no end of problems till we figured it out). This situation is tricky to troubleshoot because some parts of the network may be quite stable depending on the layout of bridges. (I had another situation once where a wireless router was reset to defaults and took on the IP address of the main gateway -- one bunch of Linux machines behind one switch was stable whereas another bunch of Windows machines behind a different switch was unstable with arp entries flip-flopping every 10 minutes or so). I mention this case in detail because: 1) you're migrating from an existing setup and there is a small chance that the old host is still on the network and has the problematic IP address, or there is some static APR entry lying around somewhere, and double checking may save you some sanity; and 2) you're using virtualization and bridges which means ARP has to flow and (in complex set ups) is why turning on STP can help. For completeness (and the archives) detecting the 'multiple hosts with the same IP' case is pretty easy: just install arpwatch on a host that receives traffic from all hosts on the network (router, dns server, dhcp server). You get a nice little email every time it thinks an IP address is flip-flopping. You also get a nice little email every time someone plugs a new device into your network (say, the cleaner's laptop at 2am). If you don't have arpwatch, then you could try: while true; do arp -n >> arp.log; sleep 5; done for 10-15 minutes (longer on a quiet network) and then do sort -u arp.log If the same IP address pops up twice, you have a problem. Tracking down the culprit is a little harder, but you can use the fact that the leading portion of the HW address is usually the same for the same NIC manufacturer. If the offending HW address is similar to those used by, say, known SUN machines, then that rules out any, say, Dell desktops. The arp.log file you just created can be handy for making these comparisons. The above problem is rare. I've only experienced on 'real' hardware 3 times in the passed 10 years, but I've seen it a lot more since I started playing with virtualization. A more common issue is ARP requests not flowing around your network properly. Bridges allow you to join networks together by passing along ARP queries. If you have multiple bridges, you can get loops, and this can prevent the ARP query reaching the right bridge, so you need STP to ensure bridges cooperate appropriately. (You can get by without STP, but only if you are 100% sure there are no loops). Firewalls can stop ARP flowing as well. On Linux this is arptables and ebtables. I note that your xen startup scripts turn this off, so you should be good, provided turning them off doesn't block ARP forwarding. Not sure about that. (I notice they are on on my systems, but I like to restrict traffic on the bridge, with the iptables/arptable/ebtables configuration managed by shorewall). I troubleshoot the above ARP issue the same way as for routing and firewalls, so lets move onto that. For newbies and the archives, I'll detail my approach. Firstly, you want to map out whether or not traffic can pass between various IP addresses. These flows may or may not be bi-directional (A to B, is not necessarily the same as B to A) depending on stateful firewalls and convoluted routing. If there are more than 2 addresses involved, things can get complicated and its easy miss important clues, so its good to be organised. I usually grab a scrap piece of paper and draw up a NxN matrix of the problematic IP addresses so I can note which flows pass and which flows fail. I also add a few good IP addresses (your desktop, routers, dhcp server, dns server, xen host, other guests, home desktop, are good candidates) so I can verify my methodology and potentially spot related issues -- these may provide clues as to the real problem. This may seem excessive, but at least you know you are getting a complete picture. I test flows with a combination of ping (or hping) and tcpdump. I normally layout 4 terminals in a 2x2 grid like so: host A: tcpdump host A: ping host B host B: tcpdump host B: ping host A For testing routes and ARP my tcpdump invocation is something like tcpdump -i ethX icmp or arp where ethX is the interface with the IP address I'm testing. For testing firewall rules I use something like tcpdump -i ethX port YYYY and invoke hping like so hping -S -p YYYY aaa.bbb.ccc.ddd where YYYY is the port you are testing and aaa.bbb.ccc.ddd the IP address. There are lots of other tcpdump options that you can use to refine what you see and cut out any irrelevant traffic (like your ssh session). Note that I only run one ping test at a time so I can watch packets flowing from both sides. The extra terminal is just to make it a little easier to be systematic in tracing reverse flows. Now if you do the above tests on a pair of IP addresses where routing and ARP is good, you should see nice little ICMP request/reply pairs and nice little ARP who-has/reply pairs. If APR isn't working, you'll be getting a strings of ARP who-has packets with no replies. If routing is broken, you get strings of ICMP requests without replies (yeah obviously) and by comparing the outputs in the three terminals, you may be able to figure out where its broken. If you fill out your flow matrix, you'll have a pretty clear picture of the situation and may have some more clues as to were the problem lies. You'll also have a nice framework for testing potential fixes. Jussi, if this is all old news to you, please forgive my rambling diatribe. From bitter experience, I just like to make sure everyone's on the same page with this stuff:-) Now lets look at some details. > Physical if cards: > Intel Corporation 82567LM-3 Gigabit Network Connection > (eth0; on motherboard) Recently I've had some onboard NICs slowly die over a period of about 2 weeks. When it happens you can get some fairly inexplicable errors. Probably not your problem, but thought I might point out the possibility. ip route output on the xen host: > 62.236.221.64/28 dev eth0 proto kernel scope link src 62.236.221.67 > 62.220.237.96/27 dev eth1 proto kernel scope link src 62.220.237.104 > default via 62.220.237.126 dev eth1 Your 'good' interface eth1 is also the default route. I gather 62.220.237.126 is your border router. Where does your 'bad' interface eth0 go then? The most common problem with routing on multi-homed systems is having packets leaving one interface and responses coming back on another. Really need to see what 'ip route' says on each of you guests. Hmmm... ifconfig on you xen host: > vif5.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > .... > vif5.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > ... > xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF > UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 Normally an interface with a 'dot' suffix is on a specific VLAN. So you've got two VLANs running here? This complicated things. Also your bridge has NOARP set which does not make much sense to me. brctl show > bridge name bridge id STP enabled interfaces > virbr0 8000.000000000000 yes > xenbr0 8000.feffffffffff no vif5.0 > vif4.0 > peth0 > vif0.0 > xenbr1 8000.feffffffffff no vif5.1 > vif3.0 > vif2.0 > peth1 > vif0.1 Okay, that makes my head hurt. Why two VLANs? What's you mapping between virtual interfaces and guests? And which guest is the bad one? Hmmm ... I could interpret this as meaning vif5.0 and vif5.1 are two interfaces on guest5 which you want to directly tie to to different physical cards. Perhaps some source based routing on your xen host would serve you better. This may be part of the overly complex setup which is causing problems. Are these guests all just web servers? Sorry if that is a too long ramble. I'm a bit confused by your set up, and as John suggested, you may need to rebuild things from scratch to sort things out. Hope this helps! Kal