I just cant get a Centos 5.1 domU to install under Centos 5.1 dom0. It is maddening! I'm trying to replace a single server that currently uses simple ethernet aliases (eth0:0, eth0:1, etc) with a new server that uses Xen virtualization.
Maybe I dont understand bridging properly or something.
My dom0 has the following: eth0 Link encap:Ethernet HWaddr 00:00:44:11:22:33 inet addr:192.168.144.6 Bcast:192.168.144.7 Mask:255.255.255.252
Notice that it is a 192.168.144.4/30 address. That is all I have for my dom0. I want two different domU's. All the examples I see online have /24 ranges.
domU #1 will be a single 192.168.139.4 domU #2 will be a subnet of 192.168.139.128/29 At this point that domU will be using 139.129, 130, and 131.
The router that I am connected to routes/forwards those subnets to my dom0.
At this stage I have modified xend-config.sxp and changed a line to (network-script /bin/true). This was based on various postings telling people to let Centos handle things instead of Xen.
A also did the following: ifcfg-eth0: DEVICE=eth0 BOOTPROTO=none NETWORK=192.168.144.4 NETMASK=255.255.255.252 IPADDR=192.168.144.6 GATEWAY=192.168.144.5 ONBOOT=yes Type=Bridge
ifcfg-xenbr0: DEVICE=xenbr0 ONBOOT=yes BRIDGE=eth0 DELAY=0 HWADDR=00:00:44:11:22:33
One problem with this is that eth0 comes up first and complains: Bringing up interface eth0: Device eth0 does not seem to be present, delaying initialization. [FAILED] Bringing up interface xenbr0: [ OK ]
So I ended up adding ifup eth0 to /etc/rc.local. What a hack. If I tried to swap xenbr0 and eth0 files (like some online examples) there are complaints about bridging operation not supported.
At this point, am I supposed to do anything special? I just run virt-manager and for shared network it only shows me eth0 (shouldn't it be xenbr0 - since they are bridged, maybe it doesnt matter)
In anaconda, I select manual/static IP address. I enter 192.168.139.4 with subnet mask of 255.255.255.255, gateway/ns of 192.168.144.5 (my router). On the next screen anaconda says that it cannot find my stage2.img. When I look at my server logs and even with tcpdump, anaconda is not making any attempt to fetch this file despite the 'unable to retrieve.' message. So there must be some networking issue between the dom0 and domU. I just have no clue. The only attempts I see via tcpdump is if I try to select DHCP server, I see the broadcast packets.
I'm not running selinux and I've tried disabling iptables as well as using 'iptables -I FORWARD -m physdev --physdev-is-bridged -j ACCEPT'
Initially from reading I thought I may want a 'routed' xen setup instead of bridged, but it seems none of the tools virt-manager, virt-install, koan, etc support 'routed'.
Ken Bass wrote on Mon, 28 Apr 2008 15:08:22 -0400:
In anaconda, I select manual/static IP address. I enter 192.168.139.4 with subnet mask of 255.255.255.255, gateway/ns of 192.168.144.5
AFAIK, there is no route from 192.168.139.4 to 192.168.144.5. You need to specify a gateway in that net! Your earlier explanation looks a bit like you think your packets somehow "go thru" your Dom0 and rely on the network settings on it. That's not the case. Once the initial bridge is setup you can use *any* IP number on the DomU that you like, there is no connection to the IP numbers that are configured for the Dom0. In other words, if you have a router with 192.168.1.1 and 192.168.2.1 and a Dom0 with 192.168.1.2 and a DomU with 192.168.2.2 you can reach that router from both with the standard (older) xen bridging that comes with CentOS 5. No problem. Also, you do not need any Iptables rules. I think you needed nothing to change in your networking setup. I have been using static IP addresses with CentOS 5/5.1 before upgrading to Xen 3.2 and they worked just fine. The only problems I had (and still have) were with DHCP.
Kai
Kai Schaetzl wrote:
Ken Bass wrote on Mon, 28 Apr 2008 15:08:22 -0400:
In anaconda, I select manual/static IP address. I enter 192.168.139.4 with subnet mask of 255.255.255.255, gateway/ns of 192.168.144.5
AFAIK, there is no route from 192.168.139.4 to 192.168.144.5. You need to specify a gateway in that net!
Hmm. I dont have a separate gateway in that net. 192.168.144.5 is the gateway for everything. But each of my ip addresses are actual real DNS entries so packets originating from them should come from them. (I used 192.168 as a private example, but my actual addresses are real). And shouldn't the installer just setup a default route to the dom0 that is installing via its eth0?
Your earlier explanation looks a bit like you think your packets somehow "go thru" your Dom0 and rely on the network settings on it. That's not the case. Once the initial bridge is setup you can use *any* IP number on the DomU that you like, there is no connection to the IP numbers that are configured for the Dom0.
That is most likely because I am confused about bridging vs routing. I realize the default is bridging. But since my ip addresses are in different subnets I cannot even get past pinging things correctly. I just don't know the proper way to correct it. Also my ip ranges, being real, are very limited so I don't have a 'pool' of ip addresses to create dummy gateways on each subnet.
I have yet to get this work. The installer will not work for me at all. I just copied a jailtime image and made my own initrd so that I could make some progress, but this bridging is still messing me up. At least with jailtime I have a shell to work with.
Ken Bass wrote on Mon, 28 Apr 2008 20:55:13 -0400:
Hmm. I dont have a separate gateway in that net. 192.168.144.5 is the gateway for everything. But each of my ip addresses are actual real DNS entries so packets originating from them should come from them.
Ok, public routable addresses. I'm not a network expert, but I don't see how these public addresses can be used if the gateway doesn't have an IP number from each of those subnets. No matter if physical machines or virtual machines are involved.
(I used
192.168 as a private example, but my actual addresses are real). And shouldn't the installer just setup a default route to the dom0 that is installing via its eth0?
No, you do not need a route there at all. Your Dom0 doesn't even need to have an IP number and your Dom0 and DomU's don't need to have IP numbers from the same net. In this regard DomU's are very much like independant physical PCs. At least if you use the standard bridging that comes with Xen.
Also my ip ranges, being real, are very limited so I don't have a 'pool' of ip addresses to create dummy gateways on each subnet.
AFAIK, your peripheral router needs one IP number from each subnet (or you need to have some other device play that role in your network and route to the router, whatever you do, it won't spare you an IP number). If you have the impression that you need an additional IP number from that subnet on the Dom0 for routing to/from the DomU and you do not want to "waste" that IP - that is not the case! (with bridging at least - if you do routing, I assume you may need that, so, with routed networking you actually need more IP numbers, I guess).
I think if your question actually revolves around "how to use as few IP addresses as necessary and have them all publically routed" this is actually not xen/vm-related at all, so you may get more answers on the main centos list. And, *do* you need to have all these machines in public subnets? For instance, I also have a public subnet for my office here and I've been using that for several years, all machines had public addresses. Recently, I wanted to free up some net space for new machines that need to be public. So, I moved all the workstations that need to have internet access, but not serve anything to a private subnet and added NAT routing on the gateway. Some PCs are now on the public subnet and some are private, and all have internet access over the same router.
Kai
-
Kai Schaetzl -
Ken Bass