[CentOS-virt] Finally switching from Xen to KVM - question about networking

Zoltan Frombach zoltan at frombach.com
Wed Jun 11 14:13:52 UTC 2014 

On 6/11/2014 4:10 PM, Steve Campbell wrote:
> On 6/11/2014 10:03 AM, Zoltan Frombach wrote:
>> On 6/11/2014 3:50 PM, Steve Campbell wrote:
>>> On 6/10/2014 4:00 PM, Zoltan Frombach wrote:
>>>> On 6/10/2014 9:51 PM, Steve Campbell wrote:
>>>>> On 6/10/2014 3:38 PM, Zoltan Frombach wrote:
>>>>>> On 6/10/2014 9:27 PM, Steve Campbell wrote:
>>>>>>> On 6/10/2014 3:09 PM, Zoltan Frombach wrote:
>>>>>>>> Steve,
>>>>>>>>
>>>>>>>> Try the following config.
>>>>>>>>
>>>>>>>> On your host:
>>>>>>>>
>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>
>>>>>>>> DEVICE=eth0
>>>>>>>> BOOTPROTO=none
>>>>>>>> ONBOOT=yes
>>>>>>>> NM_CONTROLLED=no
>>>>>>>> TYPE=Ethernet
>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- put your physical NIC's MAC address
>>>>>>>> here
>>>>>>>> BRIDGE=br0
>>>>>>>> USERCTL=no
>>>>>>>> IPV6INIT=no
>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>
>>>>>>>>
>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-br0    :
>>>>>>>>
>>>>>>>> DEVICE=br0
>>>>>>>> BOOTPROTO=static
>>>>>>>> ONBOOT=yes
>>>>>>>> NM_CONTROLLED=no
>>>>>>>> TYPE=Bridge
>>>>>>>> IPADDR=10.0.5.16
>>>>>>>> NETMASK=255.255.255.0   <- you may need to adjust this to your
>>>>>>>> network
>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>> DNS1=8.8.8.8
>>>>>>>> DNS2=8.8.4.4
>>>>>>>> PEERDNS=yes
>>>>>>>> DELAY=0
>>>>>>>> STP=off
>>>>>>>> USERCTL=no
>>>>>>>> IPV6INIT=no
>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>
>>>>>>>>
>>>>>>>> Restart networking on the host.
>>>>>>>>
>>>>>>>> Then inside you VM:
>>>>>>>>
>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>
>>>>>>>> DEVICE=eth0
>>>>>>>> BOOTPROTO=static
>>>>>>>> ONBOOT=yes
>>>>>>>> NM_CONTROLLED=no
>>>>>>>> TYPE=Ethernet
>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- it must be a unique MAC address for
>>>>>>>> your VM
>>>>>>>> IPADDR=10.0.5.17
>>>>>>>> NETMASK=255.255.255.0     <- you may need to adjust this to your
>>>>>>>> network
>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>> DNS1=8.8.8.8
>>>>>>>> DNS2=8.8.4.4
>>>>>>>> PEERDNS=yes
>>>>>>>> USERCTL=no
>>>>>>>> IPV6INIT=no
>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>
>>>>>>>>
>>>>>>>> Then in Virtual Machine Manager make sure that your VM's NIC is
>>>>>>>> connected to the br0 bridge, like this:
>>>>>>>>
>>>>>>>> Network Source: Specify shared device name
>>>>>>>>           Bridge Name: br0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 6/10/2014 8:16 PM, Steve Campbell wrote:
>>>>>>>>> On 6/10/2014 12:43 PM, Digimer wrote:
>>>>>>>>>> On 10/06/14 12:38 PM, Steve Campbell wrote:
>>>>>>>>>>> On 6/10/2014 12:05 PM, Digimer wrote:
>>>>>>>>>>>> On 10/06/14 11:46 AM, Steve Campbell wrote:
>>>>>>>>>>>>> On 6/10/2014 10:46 AM, Digimer wrote:
>>>>>>>>>>>>>> On 10/06/14 10:03 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>> I had so much trouble putting Centos 6 guest VMs on a
>>>>>>>>>>>>>>> Centos 5
>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> I finally switched to a Centos 6 host.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I've not needed more that test VMs, so I've used Virtual
>>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>> Manager
>>>>>>>>>>>>>>> on the old system, which worked pretty well, so I decided to
>>>>>>>>>>>>>>> create my
>>>>>>>>>>>>>>> first KVM guest machine. I noticed when I created it, I only
>>>>>>>>>>>>>>> had the
>>>>>>>>>>>>>>> options of NAT for my network interface, so I used that
>>>>>>>>>>>>>>> (obvious).
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Well, after starting the VM, I find I don't have
>>>>>>>>>>>>>>> connectivity
>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> interface. Reading, I find examples where I need to create
>>>>>>>>>>>>>>> bridges
>>>>>>>>>>>>>>> perhaps. Xen did most of this for me, so it's a little
>>>>>>>>>>>>>>> new to
>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can anyone throw me a clue, please?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> steve campbell
>>>>>>>>>>>>>> Setting up a bridge is not that hard, and it will give
>>>>>>>>>>>>>> your VMs
>>>>>>>>>>>>>> direct
>>>>>>>>>>>>>> access to the outside world, and host <-> VM access just
>>>>>>>>>>>>>> fine as
>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Here is a link showing how to setup a bridge connected to a
>>>>>>>>>>>>>> bond
>>>>>>>>>>>>>> device. Ignore the bond and pretend it is a straight ethX
>>>>>>>>>>>>>> device:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://alteeve.ca/w/AN!Cluster_Tutorial_2#Configuring_our_Bridge.2C_Bonds_and_Interfaces
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> The host has a device named virbr0 that is installed during
>>>>>>>>>>>>> system
>>>>>>>>>>>>> installation. It also has a network device vnet0. There are no
>>>>>>>>>>>>> files in
>>>>>>>>>>>>> /etc/sysconfig/network-scripts for these.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Shouldn't I be able to use the virbr0 virtual bridge for this?
>>>>>>>>>>>>> I've
>>>>>>>>>>>>> tried setting up the VM's device with all of the options
>>>>>>>>>>>>> that is
>>>>>>>>>>>>> listed,
>>>>>>>>>>>>> but to no avail.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Should I need to set up another bridge for this?
>>>>>>>>>>>>>
>>>>>>>>>>>>> And thanks for the link.
>>>>>>>>>>>>>
>>>>>>>>>>>>> steve
>>>>>>>>>>>> virbr0 is created and managed by libvirtd. If you open Virtual
>>>>>>>>>>>> Machine
>>>>>>>>>>>> Manager, connect to localhost and then double-click on
>>>>>>>>>>>> 'localhost',
>>>>>>>>>>>> you will see a tab for creating/managing bridges (NAT'ed,
>>>>>>>>>>>> generally).
>>>>>>>>>>>> I disable 'virbr0' as NAT'ing is generally not what I want.
>>>>>>>>>>>>
>>>>>>>>>>>> The 'vnetX' devices are dynamically created to link a VM's
>>>>>>>>>>>> interface
>>>>>>>>>>>> to a bridge. Think of them as virtual network cables. They get
>>>>>>>>>>>> created
>>>>>>>>>>>> and destroyed as needed.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> Sorry, but I'm confused:
>>>>>>>>>>>
>>>>>>>>>>> My host server has a real NIC and IP address with a real
>>>>>>>>>>> gateway to
>>>>>>>>>>> the
>>>>>>>>>>> outside:
>>>>>>>>>>> virtbr0 IP: 192.168.122.1
>>>>>>>>>>> Host IP: 10.0.5.16
>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>> on eth0 and this works
>>>>>>>>>>>
>>>>>>>>>>> My VM server has all fake stuff currently:
>>>>>>>>>>> Host IP: 10.0.5.17
>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>> on eth0 and this is like NIC without a cable.
>>>>>>>>>>>
>>>>>>>>>>> So I need to create a bridge device on both the host and VM
>>>>>>>>>>> (lets
>>>>>>>>>>> say I
>>>>>>>>>>> name it br1). I change the eth0 config file on both host and
>>>>>>>>>>> VM to
>>>>>>>>>>> point
>>>>>>>>>>> to br1 and give the br1 config file on both host and VM the
>>>>>>>>>>> correct
>>>>>>>>>>> IP.
>>>>>>>>>>> But won't this just let the two talk to each other. How will
>>>>>>>>>>> the VM
>>>>>>>>>>> server get outside?
>>>>>>>>>>>
>>>>>>>>>>> steve
>>>>>>>>>> The bridge is created only on the host. Think of the bridge as
>>>>>>>>>> being
>>>>>>>>>> like a virtual switch. When a VM is created, you tell it to
>>>>>>>>>> connect to
>>>>>>>>>> the bridge, similar to how you would plug a physical wire into a
>>>>>>>>>> real
>>>>>>>>>> switch. That provides the link to the network, and then you
>>>>>>>>>> configure
>>>>>>>>>> the virtual server's network just as if it was a real network.
>>>>>>>>>>
>>>>>>>>>> On the host, you don't set the IP on the ethX device, instead you
>>>>>>>>>> tell
>>>>>>>>>> ethX to connect to the bridge (look at 'bond2' in the link above
>>>>>>>>>> for
>>>>>>>>>> the 'BRIDGE="..."' line). Then you move the host's IP
>>>>>>>>>> address/config
>>>>>>>>>> to the bridge itself (look at the 'vbr2' in the link above).
>>>>>>>>>>
>>>>>>>>>> If you're still stuck, tell me the IP addresses you want to
>>>>>>>>>> set in
>>>>>>>>>> the
>>>>>>>>>> host and VM and I'll see if I can bang out a couple sample
>>>>>>>>>> 'ifcfg-X'
>>>>>>>>>> files.
>>>>>>>>>>
>>>>>>>>> I'm running into some problems with my config files where I get a
>>>>>>>>> message that says the network can't determine my interface names.
>>>>>>>>> I'll
>>>>>>>>> look further, but the IPs above are what I need. 10.0.5.1 is a
>>>>>>>>> firewall
>>>>>>>>> that NATs the 10.0.5.0/24 network to another outside public
>>>>>>>>> firewall.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> steve
>>>>>>>>> _______________________________________________
>>>>>>>>> CentOS-virt mailing list
>>>>>>>>> CentOS-virt at centos.org
>>>>>>>>> http://lists.centos.org/mailman/listinfo/centos-virt
>>>>>>>>>
>>>>>>> Stuck in a rut now. Restarting the network indicates :
>>>>>>>
>>>>>>> Device eth0 does not seem to be present, delaying initialization.
>>>>>>>
>>>>>>> I thought it might be due to the fact I didn't have the "TYPE" in
>>>>>>> the
>>>>>>> ifcfg-eth0 file, but after adding that, I still get the error.
>>>>>> NM_CONTROLLED=no might help with that.
>>>>> I turned off and stopped NetworkManager and had already specified
>>>>> this.
>>>>>>> I'm using the same hardware address that was in the original eth0
>>>>>>> file.
>>>>>> That is fine.
>>>>>>
>>>>>>>      My bridge device starts fine, but I'll tailor it to your specs.
>>>>>> Also, you might want to check the contents of your
>>>>>> /etc/udev/rules.d/70-persistent-net.rules file. Is the MAC address
>>>>>> for
>>>>>> eth0 the same as in your /etc/sysconfig/network-scripts/ifcfg-eth0
>>>>>> file?
>>>>>>
>>>>> They are the same except the udev file has lower case letters in it.
>>>> You mean the hexadecimal numbers are lower case? That shouldn't matter.
>>>>
>>>> Try to delete the file /etc/udev/rules.d/70-persistent-net.rules and
>>>> reboot your host machine.
>>>> ( See
>>>> http://www.andrewklau.com/device-eth0-does-not-seem-to-be-present-delaying-initialization/
>>>>
>>>> )
>>>>
>>> All is well on the host now, and I've brought up the VM and configured
>>> it to use the bridge. I can ping the host from the VM but I have no
>>> access beyond the local network. The host sees the world, but the VM
>>> only sees the 10.0.5.0/24 network.
>> Did you set the gateway in your VM to 10.0.5.1 ?
>> Can you ping the gateway ( ping 10.0.5.1 ) from the VM?
>> Do you have a firewall configured in your VM (or host) ? If so, try to
>> disable the firewall temporarily for testing.
>>
> The gateway is set. Can't ping the gateway. Firewalls (iptables and
> ip6tables) are stopped and off. Previous post suggested the route. I'll
> check it.
Add the following lines to your /etc/sysctl.conf file on your host:

net.ipv4.ip_forward = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1

then issue the following command:

sysctl -p

More information about the CentOS-virt mailing list