[CentOS-virt] Xen networking - pulling out my hair

[Ken Bass]

I just cant get a Centos 5.1 domU to install under Centos 5.1 dom0. It
is maddening! I'm trying to replace a single server that currently uses
simple ethernet aliases (eth0:0, eth0:1, etc) with a new server that
uses Xen virtualization.

Maybe I dont understand bridging properly or something.

My dom0 has the following:
eth0      Link encap:Ethernet  HWaddr 00:00:44:11:22:33
          inet addr:192.168.144.6  Bcast:192.168.144.7 Mask:255.255.255.252

Notice that it is a 192.168.144.4/30 address. That is all I have for my
dom0. I want two different domU's. All the examples I see online have
/24 ranges.

domU #1 will be a single 192.168.139.4
domU #2 will be a subnet of 192.168.139.128/29
 At this point that domU will be using 139.129, 130, and 131.

The router that I am connected to routes/forwards those subnets to my dom0.

At this stage I have modified xend-config.sxp and changed a line to
(network-script /bin/true). This was based on various postings telling
people to let Centos handle things instead of Xen.

A also did the following:
ifcfg-eth0:
DEVICE=eth0
BOOTPROTO=none
NETWORK=192.168.144.4
NETMASK=255.255.255.252
IPADDR=192.168.144.6
GATEWAY=192.168.144.5
ONBOOT=yes
Type=Bridge

ifcfg-xenbr0:
DEVICE=xenbr0
ONBOOT=yes
BRIDGE=eth0
DELAY=0
HWADDR=00:00:44:11:22:33

One problem with this is that eth0 comes up first and complains:
Bringing up interface eth0:  Device eth0 does not seem to be present,
delaying initialization.
[FAILED]
Bringing up interface xenbr0:  [  OK  ]

So I ended up adding ifup eth0 to /etc/rc.local. What a hack. If I tried
to swap xenbr0 and eth0 files (like some online examples) there are
complaints about bridging operation not supported.

At this point, am I supposed to do anything special? I just run
virt-manager and for shared network it only shows me eth0 (shouldn't it
be xenbr0 - since they are bridged, maybe it doesnt matter)

In anaconda, I select manual/static IP address. I enter 192.168.139.4
with subnet mask of 255.255.255.255, gateway/ns of 192.168.144.5 (my
router). On the next screen anaconda says that it cannot find my
stage2.img. When I look at my server logs and even with tcpdump,
anaconda is not making any attempt to fetch this file despite the
'unable to retrieve.' message. So there must be some networking issue
between the dom0 and domU. I just have no clue. The only attempts I see
via tcpdump is if I try to select DHCP server, I see the broadcast packets.

I'm not running selinux and I've tried disabling iptables as well as
using 'iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT'

Initially from reading I thought I may want a 'routed' xen setup instead
of bridged, but it seems none of the tools virt-manager, virt-install,
koan, etc support 'routed'.