[CentOS-virt] Finally switching from Xen to KVM - question about networking

Wed Jun 11 14:44:57 UTC 2014
Steve Campbell <campbell at cnpapers.com>

On 6/11/2014 10:13 AM, Zoltan Frombach wrote:
>
> On 6/11/2014 4:10 PM, Steve Campbell wrote:
>> On 6/11/2014 10:03 AM, Zoltan Frombach wrote:
>>> On 6/11/2014 3:50 PM, Steve Campbell wrote:
>>>> On 6/10/2014 4:00 PM, Zoltan Frombach wrote:
>>>>> On 6/10/2014 9:51 PM, Steve Campbell wrote:
>>>>>> On 6/10/2014 3:38 PM, Zoltan Frombach wrote:
>>>>>>> On 6/10/2014 9:27 PM, Steve Campbell wrote:
>>>>>>>> On 6/10/2014 3:09 PM, Zoltan Frombach wrote:
>>>>>>>>> Steve,
>>>>>>>>>
>>>>>>>>> Try the following config.
>>>>>>>>>
>>>>>>>>> On your host:
>>>>>>>>>
>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>
>>>>>>>>> DEVICE=eth0
>>>>>>>>> BOOTPROTO=none
>>>>>>>>> ONBOOT=yes
>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>> TYPE=Ethernet
>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- put your physical NIC's MAC 
>>>>>>>>> address
>>>>>>>>> here
>>>>>>>>> BRIDGE=br0
>>>>>>>>> USERCTL=no
>>>>>>>>> IPV6INIT=no
>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-br0    :
>>>>>>>>>
>>>>>>>>> DEVICE=br0
>>>>>>>>> BOOTPROTO=static
>>>>>>>>> ONBOOT=yes
>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>> TYPE=Bridge
>>>>>>>>> IPADDR=10.0.5.16
>>>>>>>>> NETMASK=255.255.255.0   <- you may need to adjust this to your
>>>>>>>>> network
>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>> PEERDNS=yes
>>>>>>>>> DELAY=0
>>>>>>>>> STP=off
>>>>>>>>> USERCTL=no
>>>>>>>>> IPV6INIT=no
>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Restart networking on the host.
>>>>>>>>>
>>>>>>>>> Then inside you VM:
>>>>>>>>>
>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>
>>>>>>>>> DEVICE=eth0
>>>>>>>>> BOOTPROTO=static
>>>>>>>>> ONBOOT=yes
>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>> TYPE=Ethernet
>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- it must be a unique MAC address 
>>>>>>>>> for
>>>>>>>>> your VM
>>>>>>>>> IPADDR=10.0.5.17
>>>>>>>>> NETMASK=255.255.255.0     <- you may need to adjust this to your
>>>>>>>>> network
>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>> PEERDNS=yes
>>>>>>>>> USERCTL=no
>>>>>>>>> IPV6INIT=no
>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Then in Virtual Machine Manager make sure that your VM's NIC is
>>>>>>>>> connected to the br0 bridge, like this:
>>>>>>>>>
>>>>>>>>> Network Source: Specify shared device name
>>>>>>>>>           Bridge Name: br0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 6/10/2014 8:16 PM, Steve Campbell wrote:
>>>>>>>>>> On 6/10/2014 12:43 PM, Digimer wrote:
>>>>>>>>>>> On 10/06/14 12:38 PM, Steve Campbell wrote:
>>>>>>>>>>>> On 6/10/2014 12:05 PM, Digimer wrote:
>>>>>>>>>>>>> On 10/06/14 11:46 AM, Steve Campbell wrote:
>>>>>>>>>>>>>> On 6/10/2014 10:46 AM, Digimer wrote:
>>>>>>>>>>>>>>> On 10/06/14 10:03 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>>> I had so much trouble putting Centos 6 guest VMs on a
>>>>>>>>>>>>>>>> Centos 5
>>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> I finally switched to a Centos 6 host.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I've not needed more that test VMs, so I've used Virtual
>>>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>>> Manager
>>>>>>>>>>>>>>>> on the old system, which worked pretty well, so I 
>>>>>>>>>>>>>>>> decided to
>>>>>>>>>>>>>>>> create my
>>>>>>>>>>>>>>>> first KVM guest machine. I noticed when I created it, I 
>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>> had the
>>>>>>>>>>>>>>>> options of NAT for my network interface, so I used that
>>>>>>>>>>>>>>>> (obvious).
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Well, after starting the VM, I find I don't have
>>>>>>>>>>>>>>>> connectivity
>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>> interface. Reading, I find examples where I need to create
>>>>>>>>>>>>>>>> bridges
>>>>>>>>>>>>>>>> perhaps. Xen did most of this for me, so it's a little
>>>>>>>>>>>>>>>> new to
>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Can anyone throw me a clue, please?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> steve campbell
>>>>>>>>>>>>>>> Setting up a bridge is not that hard, and it will give
>>>>>>>>>>>>>>> your VMs
>>>>>>>>>>>>>>> direct
>>>>>>>>>>>>>>> access to the outside world, and host <-> VM access just
>>>>>>>>>>>>>>> fine as
>>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Here is a link showing how to setup a bridge connected to a
>>>>>>>>>>>>>>> bond
>>>>>>>>>>>>>>> device. Ignore the bond and pretend it is a straight ethX
>>>>>>>>>>>>>>> device:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://alteeve.ca/w/AN!Cluster_Tutorial_2#Configuring_our_Bridge.2C_Bonds_and_Interfaces 
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The host has a device named virbr0 that is installed during
>>>>>>>>>>>>>> system
>>>>>>>>>>>>>> installation. It also has a network device vnet0. There 
>>>>>>>>>>>>>> are no
>>>>>>>>>>>>>> files in
>>>>>>>>>>>>>> /etc/sysconfig/network-scripts for these.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Shouldn't I be able to use the virbr0 virtual bridge for 
>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>> tried setting up the VM's device with all of the options
>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>> listed,
>>>>>>>>>>>>>> but to no avail.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Should I need to set up another bridge for this?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> And thanks for the link.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> steve
>>>>>>>>>>>>> virbr0 is created and managed by libvirtd. If you open 
>>>>>>>>>>>>> Virtual
>>>>>>>>>>>>> Machine
>>>>>>>>>>>>> Manager, connect to localhost and then double-click on
>>>>>>>>>>>>> 'localhost',
>>>>>>>>>>>>> you will see a tab for creating/managing bridges (NAT'ed,
>>>>>>>>>>>>> generally).
>>>>>>>>>>>>> I disable 'virbr0' as NAT'ing is generally not what I want.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The 'vnetX' devices are dynamically created to link a VM's
>>>>>>>>>>>>> interface
>>>>>>>>>>>>> to a bridge. Think of them as virtual network cables. They 
>>>>>>>>>>>>> get
>>>>>>>>>>>>> created
>>>>>>>>>>>>> and destroyed as needed.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Sorry, but I'm confused:
>>>>>>>>>>>>
>>>>>>>>>>>> My host server has a real NIC and IP address with a real
>>>>>>>>>>>> gateway to
>>>>>>>>>>>> the
>>>>>>>>>>>> outside:
>>>>>>>>>>>> virtbr0 IP: 192.168.122.1
>>>>>>>>>>>> Host IP: 10.0.5.16
>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>> on eth0 and this works
>>>>>>>>>>>>
>>>>>>>>>>>> My VM server has all fake stuff currently:
>>>>>>>>>>>> Host IP: 10.0.5.17
>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>> on eth0 and this is like NIC without a cable.
>>>>>>>>>>>>
>>>>>>>>>>>> So I need to create a bridge device on both the host and VM
>>>>>>>>>>>> (lets
>>>>>>>>>>>> say I
>>>>>>>>>>>> name it br1). I change the eth0 config file on both host and
>>>>>>>>>>>> VM to
>>>>>>>>>>>> point
>>>>>>>>>>>> to br1 and give the br1 config file on both host and VM the
>>>>>>>>>>>> correct
>>>>>>>>>>>> IP.
>>>>>>>>>>>> But won't this just let the two talk to each other. How will
>>>>>>>>>>>> the VM
>>>>>>>>>>>> server get outside?
>>>>>>>>>>>>
>>>>>>>>>>>> steve
>>>>>>>>>>> The bridge is created only on the host. Think of the bridge as
>>>>>>>>>>> being
>>>>>>>>>>> like a virtual switch. When a VM is created, you tell it to
>>>>>>>>>>> connect to
>>>>>>>>>>> the bridge, similar to how you would plug a physical wire 
>>>>>>>>>>> into a
>>>>>>>>>>> real
>>>>>>>>>>> switch. That provides the link to the network, and then you
>>>>>>>>>>> configure
>>>>>>>>>>> the virtual server's network just as if it was a real network.
>>>>>>>>>>>
>>>>>>>>>>> On the host, you don't set the IP on the ethX device, 
>>>>>>>>>>> instead you
>>>>>>>>>>> tell
>>>>>>>>>>> ethX to connect to the bridge (look at 'bond2' in the link 
>>>>>>>>>>> above
>>>>>>>>>>> for
>>>>>>>>>>> the 'BRIDGE="..."' line). Then you move the host's IP
>>>>>>>>>>> address/config
>>>>>>>>>>> to the bridge itself (look at the 'vbr2' in the link above).
>>>>>>>>>>>
>>>>>>>>>>> If you're still stuck, tell me the IP addresses you want to
>>>>>>>>>>> set in
>>>>>>>>>>> the
>>>>>>>>>>> host and VM and I'll see if I can bang out a couple sample
>>>>>>>>>>> 'ifcfg-X'
>>>>>>>>>>> files.
>>>>>>>>>>>
>>>>>>>>>> I'm running into some problems with my config files where I 
>>>>>>>>>> get a
>>>>>>>>>> message that says the network can't determine my interface 
>>>>>>>>>> names.
>>>>>>>>>> I'll
>>>>>>>>>> look further, but the IPs above are what I need. 10.0.5.1 is a
>>>>>>>>>> firewall
>>>>>>>>>> that NATs the 10.0.5.0/24 network to another outside public
>>>>>>>>>> firewall.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> steve
>>>>>>>>>> _______________________________________________
>>>>>>>>>> CentOS-virt mailing list
>>>>>>>>>> CentOS-virt at centos.org
>>>>>>>>>> http://lists.centos.org/mailman/listinfo/centos-virt
>>>>>>>>>>
>>>>>>>> Stuck in a rut now. Restarting the network indicates :
>>>>>>>>
>>>>>>>> Device eth0 does not seem to be present, delaying initialization.
>>>>>>>>
>>>>>>>> I thought it might be due to the fact I didn't have the "TYPE" in
>>>>>>>> the
>>>>>>>> ifcfg-eth0 file, but after adding that, I still get the error.
>>>>>>> NM_CONTROLLED=no might help with that.
>>>>>> I turned off and stopped NetworkManager and had already specified
>>>>>> this.
>>>>>>>> I'm using the same hardware address that was in the original eth0
>>>>>>>> file.
>>>>>>> That is fine.
>>>>>>>
>>>>>>>>      My bridge device starts fine, but I'll tailor it to your 
>>>>>>>> specs.
>>>>>>> Also, you might want to check the contents of your
>>>>>>> /etc/udev/rules.d/70-persistent-net.rules file. Is the MAC address
>>>>>>> for
>>>>>>> eth0 the same as in your /etc/sysconfig/network-scripts/ifcfg-eth0
>>>>>>> file?
>>>>>>>
>>>>>> They are the same except the udev file has lower case letters in it.
>>>>> You mean the hexadecimal numbers are lower case? That shouldn't 
>>>>> matter.
>>>>>
>>>>> Try to delete the file /etc/udev/rules.d/70-persistent-net.rules and
>>>>> reboot your host machine.
>>>>> ( See
>>>>> http://www.andrewklau.com/device-eth0-does-not-seem-to-be-present-delaying-initialization/ 
>>>>>
>>>>>
>>>>> )
>>>>>
>>>> All is well on the host now, and I've brought up the VM and configured
>>>> it to use the bridge. I can ping the host from the VM but I have no
>>>> access beyond the local network. The host sees the world, but the VM
>>>> only sees the 10.0.5.0/24 network.
>>> Did you set the gateway in your VM to 10.0.5.1 ?
>>> Can you ping the gateway ( ping 10.0.5.1 ) from the VM?
>>> Do you have a firewall configured in your VM (or host) ? If so, try to
>>> disable the firewall temporarily for testing.
>>>
>> The gateway is set. Can't ping the gateway. Firewalls (iptables and
>> ip6tables) are stopped and off. Previous post suggested the route. I'll
>> check it.
> Add the following lines to your /etc/sysctl.conf file on your host:
>
> net.ipv4.ip_forward = 1
> net.ipv4.conf.default.rp_filter = 0
> net.ipv6.conf.all.disable_ipv6 = 1
> net.ipv6.conf.default.disable_ipv6 = 1
>
> then issue the following command:
>
> sysctl -p
>
Still no joy. I'm installing another VM just to see if having the bridge 
defined before installation would make a difference. I find it hard to 
imagine all of this can't be done will creating the VM (Xen seemed to do 
it all for you).

So now, I've got the route, gateway, bridge, and eth0/1 defined on the 
original. I can ping both the host and the VM from each other. The host 
can see the world through the gateway. The VM can't see the gateway or 
the world.

steve