[CentOS-virt] Finally switching from Xen to KVM - question about networking

Wed Jun 11 14:56:07 UTC 2014
Zoltan Frombach <zoltan at frombach.com>

On 6/11/2014 4:44 PM, Steve Campbell wrote:
> On 6/11/2014 10:13 AM, Zoltan Frombach wrote:
>> On 6/11/2014 4:10 PM, Steve Campbell wrote:
>>> On 6/11/2014 10:03 AM, Zoltan Frombach wrote:
>>>> On 6/11/2014 3:50 PM, Steve Campbell wrote:
>>>>> On 6/10/2014 4:00 PM, Zoltan Frombach wrote:
>>>>>> On 6/10/2014 9:51 PM, Steve Campbell wrote:
>>>>>>> On 6/10/2014 3:38 PM, Zoltan Frombach wrote:
>>>>>>>> On 6/10/2014 9:27 PM, Steve Campbell wrote:
>>>>>>>>> On 6/10/2014 3:09 PM, Zoltan Frombach wrote:
>>>>>>>>>> Steve,
>>>>>>>>>>
>>>>>>>>>> Try the following config.
>>>>>>>>>>
>>>>>>>>>> On your host:
>>>>>>>>>>
>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>>
>>>>>>>>>> DEVICE=eth0
>>>>>>>>>> BOOTPROTO=none
>>>>>>>>>> ONBOOT=yes
>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>> TYPE=Ethernet
>>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- put your physical NIC's MAC
>>>>>>>>>> address
>>>>>>>>>> here
>>>>>>>>>> BRIDGE=br0
>>>>>>>>>> USERCTL=no
>>>>>>>>>> IPV6INIT=no
>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-br0    :
>>>>>>>>>>
>>>>>>>>>> DEVICE=br0
>>>>>>>>>> BOOTPROTO=static
>>>>>>>>>> ONBOOT=yes
>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>> TYPE=Bridge
>>>>>>>>>> IPADDR=10.0.5.16
>>>>>>>>>> NETMASK=255.255.255.0   <- you may need to adjust this to your
>>>>>>>>>> network
>>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>>> PEERDNS=yes
>>>>>>>>>> DELAY=0
>>>>>>>>>> STP=off
>>>>>>>>>> USERCTL=no
>>>>>>>>>> IPV6INIT=no
>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Restart networking on the host.
>>>>>>>>>>
>>>>>>>>>> Then inside you VM:
>>>>>>>>>>
>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>>
>>>>>>>>>> DEVICE=eth0
>>>>>>>>>> BOOTPROTO=static
>>>>>>>>>> ONBOOT=yes
>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>> TYPE=Ethernet
>>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- it must be a unique MAC address
>>>>>>>>>> for
>>>>>>>>>> your VM
>>>>>>>>>> IPADDR=10.0.5.17
>>>>>>>>>> NETMASK=255.255.255.0     <- you may need to adjust this to your
>>>>>>>>>> network
>>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>>> PEERDNS=yes
>>>>>>>>>> USERCTL=no
>>>>>>>>>> IPV6INIT=no
>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Then in Virtual Machine Manager make sure that your VM's NIC is
>>>>>>>>>> connected to the br0 bridge, like this:
>>>>>>>>>>
>>>>>>>>>> Network Source: Specify shared device name
>>>>>>>>>>            Bridge Name: br0
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 6/10/2014 8:16 PM, Steve Campbell wrote:
>>>>>>>>>>> On 6/10/2014 12:43 PM, Digimer wrote:
>>>>>>>>>>>> On 10/06/14 12:38 PM, Steve Campbell wrote:
>>>>>>>>>>>>> On 6/10/2014 12:05 PM, Digimer wrote:
>>>>>>>>>>>>>> On 10/06/14 11:46 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>> On 6/10/2014 10:46 AM, Digimer wrote:
>>>>>>>>>>>>>>>> On 10/06/14 10:03 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>>>> I had so much trouble putting Centos 6 guest VMs on a
>>>>>>>>>>>>>>>>> Centos 5
>>>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> I finally switched to a Centos 6 host.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I've not needed more that test VMs, so I've used Virtual
>>>>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>>>> Manager
>>>>>>>>>>>>>>>>> on the old system, which worked pretty well, so I
>>>>>>>>>>>>>>>>> decided to
>>>>>>>>>>>>>>>>> create my
>>>>>>>>>>>>>>>>> first KVM guest machine. I noticed when I created it, I
>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>> had the
>>>>>>>>>>>>>>>>> options of NAT for my network interface, so I used that
>>>>>>>>>>>>>>>>> (obvious).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Well, after starting the VM, I find I don't have
>>>>>>>>>>>>>>>>> connectivity
>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>> interface. Reading, I find examples where I need to create
>>>>>>>>>>>>>>>>> bridges
>>>>>>>>>>>>>>>>> perhaps. Xen did most of this for me, so it's a little
>>>>>>>>>>>>>>>>> new to
>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Can anyone throw me a clue, please?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> steve campbell
>>>>>>>>>>>>>>>> Setting up a bridge is not that hard, and it will give
>>>>>>>>>>>>>>>> your VMs
>>>>>>>>>>>>>>>> direct
>>>>>>>>>>>>>>>> access to the outside world, and host <-> VM access just
>>>>>>>>>>>>>>>> fine as
>>>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Here is a link showing how to setup a bridge connected to a
>>>>>>>>>>>>>>>> bond
>>>>>>>>>>>>>>>> device. Ignore the bond and pretend it is a straight ethX
>>>>>>>>>>>>>>>> device:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> https://alteeve.ca/w/AN!Cluster_Tutorial_2#Configuring_our_Bridge.2C_Bonds_and_Interfaces
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The host has a device named virbr0 that is installed during
>>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>> installation. It also has a network device vnet0. There
>>>>>>>>>>>>>>> are no
>>>>>>>>>>>>>>> files in
>>>>>>>>>>>>>>> /etc/sysconfig/network-scripts for these.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Shouldn't I be able to use the virbr0 virtual bridge for
>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>> tried setting up the VM's device with all of the options
>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>> listed,
>>>>>>>>>>>>>>> but to no avail.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Should I need to set up another bridge for this?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> And thanks for the link.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> steve
>>>>>>>>>>>>>> virbr0 is created and managed by libvirtd. If you open
>>>>>>>>>>>>>> Virtual
>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>> Manager, connect to localhost and then double-click on
>>>>>>>>>>>>>> 'localhost',
>>>>>>>>>>>>>> you will see a tab for creating/managing bridges (NAT'ed,
>>>>>>>>>>>>>> generally).
>>>>>>>>>>>>>> I disable 'virbr0' as NAT'ing is generally not what I want.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The 'vnetX' devices are dynamically created to link a VM's
>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>> to a bridge. Think of them as virtual network cables. They
>>>>>>>>>>>>>> get
>>>>>>>>>>>>>> created
>>>>>>>>>>>>>> and destroyed as needed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Sorry, but I'm confused:
>>>>>>>>>>>>>
>>>>>>>>>>>>> My host server has a real NIC and IP address with a real
>>>>>>>>>>>>> gateway to
>>>>>>>>>>>>> the
>>>>>>>>>>>>> outside:
>>>>>>>>>>>>> virtbr0 IP: 192.168.122.1
>>>>>>>>>>>>> Host IP: 10.0.5.16
>>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>>> on eth0 and this works
>>>>>>>>>>>>>
>>>>>>>>>>>>> My VM server has all fake stuff currently:
>>>>>>>>>>>>> Host IP: 10.0.5.17
>>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>>> on eth0 and this is like NIC without a cable.
>>>>>>>>>>>>>
>>>>>>>>>>>>> So I need to create a bridge device on both the host and VM
>>>>>>>>>>>>> (lets
>>>>>>>>>>>>> say I
>>>>>>>>>>>>> name it br1). I change the eth0 config file on both host and
>>>>>>>>>>>>> VM to
>>>>>>>>>>>>> point
>>>>>>>>>>>>> to br1 and give the br1 config file on both host and VM the
>>>>>>>>>>>>> correct
>>>>>>>>>>>>> IP.
>>>>>>>>>>>>> But won't this just let the two talk to each other. How will
>>>>>>>>>>>>> the VM
>>>>>>>>>>>>> server get outside?
>>>>>>>>>>>>>
>>>>>>>>>>>>> steve
>>>>>>>>>>>> The bridge is created only on the host. Think of the bridge as
>>>>>>>>>>>> being
>>>>>>>>>>>> like a virtual switch. When a VM is created, you tell it to
>>>>>>>>>>>> connect to
>>>>>>>>>>>> the bridge, similar to how you would plug a physical wire
>>>>>>>>>>>> into a
>>>>>>>>>>>> real
>>>>>>>>>>>> switch. That provides the link to the network, and then you
>>>>>>>>>>>> configure
>>>>>>>>>>>> the virtual server's network just as if it was a real network.
>>>>>>>>>>>>
>>>>>>>>>>>> On the host, you don't set the IP on the ethX device,
>>>>>>>>>>>> instead you
>>>>>>>>>>>> tell
>>>>>>>>>>>> ethX to connect to the bridge (look at 'bond2' in the link
>>>>>>>>>>>> above
>>>>>>>>>>>> for
>>>>>>>>>>>> the 'BRIDGE="..."' line). Then you move the host's IP
>>>>>>>>>>>> address/config
>>>>>>>>>>>> to the bridge itself (look at the 'vbr2' in the link above).
>>>>>>>>>>>>
>>>>>>>>>>>> If you're still stuck, tell me the IP addresses you want to
>>>>>>>>>>>> set in
>>>>>>>>>>>> the
>>>>>>>>>>>> host and VM and I'll see if I can bang out a couple sample
>>>>>>>>>>>> 'ifcfg-X'
>>>>>>>>>>>> files.
>>>>>>>>>>>>
>>>>>>>>>>> I'm running into some problems with my config files where I
>>>>>>>>>>> get a
>>>>>>>>>>> message that says the network can't determine my interface
>>>>>>>>>>> names.
>>>>>>>>>>> I'll
>>>>>>>>>>> look further, but the IPs above are what I need. 10.0.5.1 is a
>>>>>>>>>>> firewall
>>>>>>>>>>> that NATs the 10.0.5.0/24 network to another outside public
>>>>>>>>>>> firewall.
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> steve
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> CentOS-virt mailing list
>>>>>>>>>>> CentOS-virt at centos.org
>>>>>>>>>>> http://lists.centos.org/mailman/listinfo/centos-virt
>>>>>>>>>>>
>>>>>>>>> Stuck in a rut now. Restarting the network indicates :
>>>>>>>>>
>>>>>>>>> Device eth0 does not seem to be present, delaying initialization.
>>>>>>>>>
>>>>>>>>> I thought it might be due to the fact I didn't have the "TYPE" in
>>>>>>>>> the
>>>>>>>>> ifcfg-eth0 file, but after adding that, I still get the error.
>>>>>>>> NM_CONTROLLED=no might help with that.
>>>>>>> I turned off and stopped NetworkManager and had already specified
>>>>>>> this.
>>>>>>>>> I'm using the same hardware address that was in the original eth0
>>>>>>>>> file.
>>>>>>>> That is fine.
>>>>>>>>
>>>>>>>>>       My bridge device starts fine, but I'll tailor it to your
>>>>>>>>> specs.
>>>>>>>> Also, you might want to check the contents of your
>>>>>>>> /etc/udev/rules.d/70-persistent-net.rules file. Is the MAC address
>>>>>>>> for
>>>>>>>> eth0 the same as in your /etc/sysconfig/network-scripts/ifcfg-eth0
>>>>>>>> file?
>>>>>>>>
>>>>>>> They are the same except the udev file has lower case letters in it.
>>>>>> You mean the hexadecimal numbers are lower case? That shouldn't
>>>>>> matter.
>>>>>>
>>>>>> Try to delete the file /etc/udev/rules.d/70-persistent-net.rules and
>>>>>> reboot your host machine.
>>>>>> ( See
>>>>>> http://www.andrewklau.com/device-eth0-does-not-seem-to-be-present-delaying-initialization/
>>>>>>
>>>>>>
>>>>>> )
>>>>>>
>>>>> All is well on the host now, and I've brought up the VM and configured
>>>>> it to use the bridge. I can ping the host from the VM but I have no
>>>>> access beyond the local network. The host sees the world, but the VM
>>>>> only sees the 10.0.5.0/24 network.
>>>> Did you set the gateway in your VM to 10.0.5.1 ?
>>>> Can you ping the gateway ( ping 10.0.5.1 ) from the VM?
>>>> Do you have a firewall configured in your VM (or host) ? If so, try to
>>>> disable the firewall temporarily for testing.
>>>>
>>> The gateway is set. Can't ping the gateway. Firewalls (iptables and
>>> ip6tables) are stopped and off. Previous post suggested the route. I'll
>>> check it.
>> Add the following lines to your /etc/sysctl.conf file on your host:
>>
>> net.ipv4.ip_forward = 1
>> net.ipv4.conf.default.rp_filter = 0
>> net.ipv6.conf.all.disable_ipv6 = 1
>> net.ipv6.conf.default.disable_ipv6 = 1
>>
>> then issue the following command:
>>
>> sysctl -p
>>
> Still no joy. I'm installing another VM just to see if having the bridge
> defined before installation would make a difference. I find it hard to
> imagine all of this can't be done will creating the VM (Xen seemed to do
> it all for you).

What you did on the host machine, only has to be done once when you 
setup the system to be used as a KVM host server.

> So now, I've got the route, gateway, bridge, and eth0/1 defined on the
> original. I can ping both the host and the VM from each other. The host
> can see the world through the gateway. The VM can't see the gateway or
> the world.

How did you come up with the IP address ( 10.0.5.17 ) for the VM ? Are 
you sure it is not blocked somehow by your gateway ( at 10.0.5.1 ) ?

Also, maybe your gateway ( at 10.0.5.1 ) is doing MAC address filtering? 
Since your VM has its own MAC address, this could be a problem.