[CentOS-virt] Finally switching from Xen to KVM - question about networking

Wed Jun 11 17:37:49 UTC 2014
Zoltan Frombach <zoltan at frombach.com>

On 6/11/2014 7:27 PM, Steve Campbell wrote:
> On 6/11/2014 1:15 PM, Steve Campbell wrote:
>> On 6/11/2014 10:56 AM, Zoltan Frombach wrote:
>>> On 6/11/2014 4:44 PM, Steve Campbell wrote:
>>>> On 6/11/2014 10:13 AM, Zoltan Frombach wrote:
>>>>> On 6/11/2014 4:10 PM, Steve Campbell wrote:
>>>>>> On 6/11/2014 10:03 AM, Zoltan Frombach wrote:
>>>>>>> On 6/11/2014 3:50 PM, Steve Campbell wrote:
>>>>>>>> On 6/10/2014 4:00 PM, Zoltan Frombach wrote:
>>>>>>>>> On 6/10/2014 9:51 PM, Steve Campbell wrote:
>>>>>>>>>> On 6/10/2014 3:38 PM, Zoltan Frombach wrote:
>>>>>>>>>>> On 6/10/2014 9:27 PM, Steve Campbell wrote:
>>>>>>>>>>>> On 6/10/2014 3:09 PM, Zoltan Frombach wrote:
>>>>>>>>>>>>> Steve,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Try the following config.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On your host:
>>>>>>>>>>>>>
>>>>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>>>>>
>>>>>>>>>>>>> DEVICE=eth0
>>>>>>>>>>>>> BOOTPROTO=none
>>>>>>>>>>>>> ONBOOT=yes
>>>>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>>>>> TYPE=Ethernet
>>>>>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- put your physical NIC's MAC
>>>>>>>>>>>>> address
>>>>>>>>>>>>> here
>>>>>>>>>>>>> BRIDGE=br0
>>>>>>>>>>>>> USERCTL=no
>>>>>>>>>>>>> IPV6INIT=no
>>>>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-br0    :
>>>>>>>>>>>>>
>>>>>>>>>>>>> DEVICE=br0
>>>>>>>>>>>>> BOOTPROTO=static
>>>>>>>>>>>>> ONBOOT=yes
>>>>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>>>>> TYPE=Bridge
>>>>>>>>>>>>> IPADDR=10.0.5.16
>>>>>>>>>>>>> NETMASK=255.255.255.0   <- you may need to adjust this to your
>>>>>>>>>>>>> network
>>>>>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>>>>>> PEERDNS=yes
>>>>>>>>>>>>> DELAY=0
>>>>>>>>>>>>> STP=off
>>>>>>>>>>>>> USERCTL=no
>>>>>>>>>>>>> IPV6INIT=no
>>>>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Restart networking on the host.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then inside you VM:
>>>>>>>>>>>>>
>>>>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>>>>>
>>>>>>>>>>>>> DEVICE=eth0
>>>>>>>>>>>>> BOOTPROTO=static
>>>>>>>>>>>>> ONBOOT=yes
>>>>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>>>>> TYPE=Ethernet
>>>>>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- it must be a unique MAC address
>>>>>>>>>>>>> for
>>>>>>>>>>>>> your VM
>>>>>>>>>>>>> IPADDR=10.0.5.17
>>>>>>>>>>>>> NETMASK=255.255.255.0     <- you may need to adjust this to
>>>>>>>>>>>>> your
>>>>>>>>>>>>> network
>>>>>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>>>>>> PEERDNS=yes
>>>>>>>>>>>>> USERCTL=no
>>>>>>>>>>>>> IPV6INIT=no
>>>>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then in Virtual Machine Manager make sure that your VM's NIC is
>>>>>>>>>>>>> connected to the br0 bridge, like this:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Network Source: Specify shared device name
>>>>>>>>>>>>>              Bridge Name: br0
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 6/10/2014 8:16 PM, Steve Campbell wrote:
>>>>>>>>>>>>>> On 6/10/2014 12:43 PM, Digimer wrote:
>>>>>>>>>>>>>>> On 10/06/14 12:38 PM, Steve Campbell wrote:
>>>>>>>>>>>>>>>> On 6/10/2014 12:05 PM, Digimer wrote:
>>>>>>>>>>>>>>>>> On 10/06/14 11:46 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>>>>> On 6/10/2014 10:46 AM, Digimer wrote:
>>>>>>>>>>>>>>>>>>> On 10/06/14 10:03 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>>>>>>> I had so much trouble putting Centos 6 guest VMs on a
>>>>>>>>>>>>>>>>>>>> Centos 5
>>>>>>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> I finally switched to a Centos 6 host.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I've not needed more that test VMs, so I've used Virtual
>>>>>>>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>>>>>>> Manager
>>>>>>>>>>>>>>>>>>>> on the old system, which worked pretty well, so I
>>>>>>>>>>>>>>>>>>>> decided to
>>>>>>>>>>>>>>>>>>>> create my
>>>>>>>>>>>>>>>>>>>> first KVM guest machine. I noticed when I created it, I
>>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>>> had the
>>>>>>>>>>>>>>>>>>>> options of NAT for my network interface, so I used that
>>>>>>>>>>>>>>>>>>>> (obvious).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Well, after starting the VM, I find I don't have
>>>>>>>>>>>>>>>>>>>> connectivity
>>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>>> interface. Reading, I find examples where I need to
>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>> bridges
>>>>>>>>>>>>>>>>>>>> perhaps. Xen did most of this for me, so it's a little
>>>>>>>>>>>>>>>>>>>> new to
>>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Can anyone throw me a clue, please?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> steve campbell
>>>>>>>>>>>>>>>>>>> Setting up a bridge is not that hard, and it will give
>>>>>>>>>>>>>>>>>>> your VMs
>>>>>>>>>>>>>>>>>>> direct
>>>>>>>>>>>>>>>>>>> access to the outside world, and host <-> VM access just
>>>>>>>>>>>>>>>>>>> fine as
>>>>>>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Here is a link showing how to setup a bridge connected
>>>>>>>>>>>>>>>>>>> to a
>>>>>>>>>>>>>>>>>>> bond
>>>>>>>>>>>>>>>>>>> device. Ignore the bond and pretend it is a straight ethX
>>>>>>>>>>>>>>>>>>> device:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> https://alteeve.ca/w/AN!Cluster_Tutorial_2#Configuring_our_Bridge.2C_Bonds_and_Interfaces
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> The host has a device named virbr0 that is installed
>>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>>>>> installation. It also has a network device vnet0. There
>>>>>>>>>>>>>>>>>> are no
>>>>>>>>>>>>>>>>>> files in
>>>>>>>>>>>>>>>>>> /etc/sysconfig/network-scripts for these.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Shouldn't I be able to use the virbr0 virtual bridge for
>>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>>> tried setting up the VM's device with all of the options
>>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>>> listed,
>>>>>>>>>>>>>>>>>> but to no avail.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Should I need to set up another bridge for this?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> And thanks for the link.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> steve
>>>>>>>>>>>>>>>>> virbr0 is created and managed by libvirtd. If you open
>>>>>>>>>>>>>>>>> Virtual
>>>>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>>>> Manager, connect to localhost and then double-click on
>>>>>>>>>>>>>>>>> 'localhost',
>>>>>>>>>>>>>>>>> you will see a tab for creating/managing bridges (NAT'ed,
>>>>>>>>>>>>>>>>> generally).
>>>>>>>>>>>>>>>>> I disable 'virbr0' as NAT'ing is generally not what I want.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The 'vnetX' devices are dynamically created to link a VM's
>>>>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>>> to a bridge. Think of them as virtual network cables. They
>>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>>> and destroyed as needed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sorry, but I'm confused:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My host server has a real NIC and IP address with a real
>>>>>>>>>>>>>>>> gateway to
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>> outside:
>>>>>>>>>>>>>>>> virtbr0 IP: 192.168.122.1
>>>>>>>>>>>>>>>> Host IP: 10.0.5.16
>>>>>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>>>>>> on eth0 and this works
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> My VM server has all fake stuff currently:
>>>>>>>>>>>>>>>> Host IP: 10.0.5.17
>>>>>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>>>>>> on eth0 and this is like NIC without a cable.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> So I need to create a bridge device on both the host and VM
>>>>>>>>>>>>>>>> (lets
>>>>>>>>>>>>>>>> say I
>>>>>>>>>>>>>>>> name it br1). I change the eth0 config file on both host and
>>>>>>>>>>>>>>>> VM to
>>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>>> to br1 and give the br1 config file on both host and VM the
>>>>>>>>>>>>>>>> correct
>>>>>>>>>>>>>>>> IP.
>>>>>>>>>>>>>>>> But won't this just let the two talk to each other. How will
>>>>>>>>>>>>>>>> the VM
>>>>>>>>>>>>>>>> server get outside?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> steve
>>>>>>>>>>>>>>> The bridge is created only on the host. Think of the
>>>>>>>>>>>>>>> bridge as
>>>>>>>>>>>>>>> being
>>>>>>>>>>>>>>> like a virtual switch. When a VM is created, you tell it to
>>>>>>>>>>>>>>> connect to
>>>>>>>>>>>>>>> the bridge, similar to how you would plug a physical wire
>>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>>> real
>>>>>>>>>>>>>>> switch. That provides the link to the network, and then you
>>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>>> the virtual server's network just as if it was a real
>>>>>>>>>>>>>>> network.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On the host, you don't set the IP on the ethX device,
>>>>>>>>>>>>>>> instead you
>>>>>>>>>>>>>>> tell
>>>>>>>>>>>>>>> ethX to connect to the bridge (look at 'bond2' in the link
>>>>>>>>>>>>>>> above
>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>> the 'BRIDGE="..."' line). Then you move the host's IP
>>>>>>>>>>>>>>> address/config
>>>>>>>>>>>>>>> to the bridge itself (look at the 'vbr2' in the link above).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you're still stuck, tell me the IP addresses you want to
>>>>>>>>>>>>>>> set in
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> host and VM and I'll see if I can bang out a couple sample
>>>>>>>>>>>>>>> 'ifcfg-X'
>>>>>>>>>>>>>>> files.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'm running into some problems with my config files where I
>>>>>>>>>>>>>> get a
>>>>>>>>>>>>>> message that says the network can't determine my interface
>>>>>>>>>>>>>> names.
>>>>>>>>>>>>>> I'll
>>>>>>>>>>>>>> look further, but the IPs above are what I need. 10.0.5.1 is a
>>>>>>>>>>>>>> firewall
>>>>>>>>>>>>>> that NATs the 10.0.5.0/24 network to another outside public
>>>>>>>>>>>>>> firewall.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> steve
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> CentOS-virt mailing list
>>>>>>>>>>>>>> CentOS-virt at centos.org
>>>>>>>>>>>>>> http://lists.centos.org/mailman/listinfo/centos-virt
>>>>>>>>>>>>>>
>>>>>>>>>>>> Stuck in a rut now. Restarting the network indicates :
>>>>>>>>>>>>
>>>>>>>>>>>> Device eth0 does not seem to be present, delaying
>>>>>>>>>>>> initialization.
>>>>>>>>>>>>
>>>>>>>>>>>> I thought it might be due to the fact I didn't have the
>>>>>>>>>>>> "TYPE" in
>>>>>>>>>>>> the
>>>>>>>>>>>> ifcfg-eth0 file, but after adding that, I still get the error.
>>>>>>>>>>> NM_CONTROLLED=no might help with that.
>>>>>>>>>> I turned off and stopped NetworkManager and had already specified
>>>>>>>>>> this.
>>>>>>>>>>>> I'm using the same hardware address that was in the original
>>>>>>>>>>>> eth0
>>>>>>>>>>>> file.
>>>>>>>>>>> That is fine.
>>>>>>>>>>>
>>>>>>>>>>>>         My bridge device starts fine, but I'll tailor it to your
>>>>>>>>>>>> specs.
>>>>>>>>>>> Also, you might want to check the contents of your
>>>>>>>>>>> /etc/udev/rules.d/70-persistent-net.rules file. Is the MAC
>>>>>>>>>>> address
>>>>>>>>>>> for
>>>>>>>>>>> eth0 the same as in your
>>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0
>>>>>>>>>>> file?
>>>>>>>>>>>
>>>>>>>>>> They are the same except the udev file has lower case letters
>>>>>>>>>> in it.
>>>>>>>>> You mean the hexadecimal numbers are lower case? That shouldn't
>>>>>>>>> matter.
>>>>>>>>>
>>>>>>>>> Try to delete the file /etc/udev/rules.d/70-persistent-net.rules
>>>>>>>>> and
>>>>>>>>> reboot your host machine.
>>>>>>>>> ( See
>>>>>>>>> http://www.andrewklau.com/device-eth0-does-not-seem-to-be-present-delaying-initialization/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> )
>>>>>>>>>
>>>>>>>> All is well on the host now, and I've brought up the VM and
>>>>>>>> configured
>>>>>>>> it to use the bridge. I can ping the host from the VM but I have no
>>>>>>>> access beyond the local network. The host sees the world, but the VM
>>>>>>>> only sees the 10.0.5.0/24 network.
>>>>>>> Did you set the gateway in your VM to 10.0.5.1 ?
>>>>>>> Can you ping the gateway ( ping 10.0.5.1 ) from the VM?
>>>>>>> Do you have a firewall configured in your VM (or host) ? If so,
>>>>>>> try to
>>>>>>> disable the firewall temporarily for testing.
>>>>>>>
>>>>>> The gateway is set. Can't ping the gateway. Firewalls (iptables and
>>>>>> ip6tables) are stopped and off. Previous post suggested the route.
>>>>>> I'll
>>>>>> check it.
>>>>> Add the following lines to your /etc/sysctl.conf file on your host:
>>>>>
>>>>> net.ipv4.ip_forward = 1
>>>>> net.ipv4.conf.default.rp_filter = 0
>>>>> net.ipv6.conf.all.disable_ipv6 = 1
>>>>> net.ipv6.conf.default.disable_ipv6 = 1
>>>>>
>>>>> then issue the following command:
>>>>>
>>>>> sysctl -p
>>>>>
>>>> Still no joy. I'm installing another VM just to see if having the bridge
>>>> defined before installation would make a difference. I find it hard to
>>>> imagine all of this can't be done will creating the VM (Xen seemed to do
>>>> it all for you).
>>> What you did on the host machine, only has to be done once when you
>>> setup the system to be used as a KVM host server.
>>>
>>>> So now, I've got the route, gateway, bridge, and eth0/1 defined on the
>>>> original. I can ping both the host and the VM from each other. The host
>>>> can see the world through the gateway. The VM can't see the gateway or
>>>> the world.
>>> How did you come up with the IP address ( 10.0.5.17 ) for the VM ? Are
>>> you sure it is not blocked somehow by your gateway ( at 10.0.5.1 ) ?
>>>
>>> Also, maybe your gateway ( at 10.0.5.1 ) is doing MAC address
>>> filtering? Since your VM has its own MAC address, this could be a
>>> problem.
>>>
>> The gateway is a firewall. It's set up to route and NAT from 10.0.5.0/24
>> to another inside network on 10.0.0.0/24. This is the true inside
>> network that is NATted to the public with our public IPs. It accepts all
>> traffic from the 10.0.5.0 network.
>>
>>
> I created another VM, this time the bridge was available when it was
> created, an it's working. I'll delete the first VM and re-create it and
> see if that doesn't resolve it.
>
> Thanks for all the help and tutoring on this.

Glad to hear that it works!