[CentOS-virt] Finally switching from Xen to KVM - question about networking

Wed Jun 11 17:27:52 UTC 2014
Steve Campbell <campbell at cnpapers.com>

On 6/11/2014 1:15 PM, Steve Campbell wrote:
> On 6/11/2014 10:56 AM, Zoltan Frombach wrote:
>> On 6/11/2014 4:44 PM, Steve Campbell wrote:
>>> On 6/11/2014 10:13 AM, Zoltan Frombach wrote:
>>>> On 6/11/2014 4:10 PM, Steve Campbell wrote:
>>>>> On 6/11/2014 10:03 AM, Zoltan Frombach wrote:
>>>>>> On 6/11/2014 3:50 PM, Steve Campbell wrote:
>>>>>>> On 6/10/2014 4:00 PM, Zoltan Frombach wrote:
>>>>>>>> On 6/10/2014 9:51 PM, Steve Campbell wrote:
>>>>>>>>> On 6/10/2014 3:38 PM, Zoltan Frombach wrote:
>>>>>>>>>> On 6/10/2014 9:27 PM, Steve Campbell wrote:
>>>>>>>>>>> On 6/10/2014 3:09 PM, Zoltan Frombach wrote:
>>>>>>>>>>>> Steve,
>>>>>>>>>>>>
>>>>>>>>>>>> Try the following config.
>>>>>>>>>>>>
>>>>>>>>>>>> On your host:
>>>>>>>>>>>>
>>>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>>>>
>>>>>>>>>>>> DEVICE=eth0
>>>>>>>>>>>> BOOTPROTO=none
>>>>>>>>>>>> ONBOOT=yes
>>>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>>>> TYPE=Ethernet
>>>>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- put your physical NIC's MAC
>>>>>>>>>>>> address
>>>>>>>>>>>> here
>>>>>>>>>>>> BRIDGE=br0
>>>>>>>>>>>> USERCTL=no
>>>>>>>>>>>> IPV6INIT=no
>>>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-br0    :
>>>>>>>>>>>>
>>>>>>>>>>>> DEVICE=br0
>>>>>>>>>>>> BOOTPROTO=static
>>>>>>>>>>>> ONBOOT=yes
>>>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>>>> TYPE=Bridge
>>>>>>>>>>>> IPADDR=10.0.5.16
>>>>>>>>>>>> NETMASK=255.255.255.0   <- you may need to adjust this to your
>>>>>>>>>>>> network
>>>>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>>>>> PEERDNS=yes
>>>>>>>>>>>> DELAY=0
>>>>>>>>>>>> STP=off
>>>>>>>>>>>> USERCTL=no
>>>>>>>>>>>> IPV6INIT=no
>>>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Restart networking on the host.
>>>>>>>>>>>>
>>>>>>>>>>>> Then inside you VM:
>>>>>>>>>>>>
>>>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0    :
>>>>>>>>>>>>
>>>>>>>>>>>> DEVICE=eth0
>>>>>>>>>>>> BOOTPROTO=static
>>>>>>>>>>>> ONBOOT=yes
>>>>>>>>>>>> NM_CONTROLLED=no
>>>>>>>>>>>> TYPE=Ethernet
>>>>>>>>>>>> HWADDR=xx:xx:xx:xx:xx:xx    <- it must be a unique MAC address
>>>>>>>>>>>> for
>>>>>>>>>>>> your VM
>>>>>>>>>>>> IPADDR=10.0.5.17
>>>>>>>>>>>> NETMASK=255.255.255.0     <- you may need to adjust this to
>>>>>>>>>>>> your
>>>>>>>>>>>> network
>>>>>>>>>>>> GATEWAY=10.0.5.1
>>>>>>>>>>>> DNS1=8.8.8.8
>>>>>>>>>>>> DNS2=8.8.4.4
>>>>>>>>>>>> PEERDNS=yes
>>>>>>>>>>>> USERCTL=no
>>>>>>>>>>>> IPV6INIT=no
>>>>>>>>>>>> IPV6_AUTOCONF=no
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Then in Virtual Machine Manager make sure that your VM's NIC is
>>>>>>>>>>>> connected to the br0 bridge, like this:
>>>>>>>>>>>>
>>>>>>>>>>>> Network Source: Specify shared device name
>>>>>>>>>>>>             Bridge Name: br0
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 6/10/2014 8:16 PM, Steve Campbell wrote:
>>>>>>>>>>>>> On 6/10/2014 12:43 PM, Digimer wrote:
>>>>>>>>>>>>>> On 10/06/14 12:38 PM, Steve Campbell wrote:
>>>>>>>>>>>>>>> On 6/10/2014 12:05 PM, Digimer wrote:
>>>>>>>>>>>>>>>> On 10/06/14 11:46 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>>>> On 6/10/2014 10:46 AM, Digimer wrote:
>>>>>>>>>>>>>>>>>> On 10/06/14 10:03 AM, Steve Campbell wrote:
>>>>>>>>>>>>>>>>>>> I had so much trouble putting Centos 6 guest VMs on a
>>>>>>>>>>>>>>>>>>> Centos 5
>>>>>>>>>>>>>>>>>>> host
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> I finally switched to a Centos 6 host.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I've not needed more that test VMs, so I've used Virtual
>>>>>>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>>>>>> Manager
>>>>>>>>>>>>>>>>>>> on the old system, which worked pretty well, so I
>>>>>>>>>>>>>>>>>>> decided to
>>>>>>>>>>>>>>>>>>> create my
>>>>>>>>>>>>>>>>>>> first KVM guest machine. I noticed when I created it, I
>>>>>>>>>>>>>>>>>>> only
>>>>>>>>>>>>>>>>>>> had the
>>>>>>>>>>>>>>>>>>> options of NAT for my network interface, so I used that
>>>>>>>>>>>>>>>>>>> (obvious).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Well, after starting the VM, I find I don't have
>>>>>>>>>>>>>>>>>>> connectivity
>>>>>>>>>>>>>>>>>>> with
>>>>>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>>>>>> interface. Reading, I find examples where I need to
>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>> bridges
>>>>>>>>>>>>>>>>>>> perhaps. Xen did most of this for me, so it's a little
>>>>>>>>>>>>>>>>>>> new to
>>>>>>>>>>>>>>>>>>> me.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Can anyone throw me a clue, please?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> steve campbell
>>>>>>>>>>>>>>>>>> Setting up a bridge is not that hard, and it will give
>>>>>>>>>>>>>>>>>> your VMs
>>>>>>>>>>>>>>>>>> direct
>>>>>>>>>>>>>>>>>> access to the outside world, and host <-> VM access just
>>>>>>>>>>>>>>>>>> fine as
>>>>>>>>>>>>>>>>>> well.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Here is a link showing how to setup a bridge connected
>>>>>>>>>>>>>>>>>> to a
>>>>>>>>>>>>>>>>>> bond
>>>>>>>>>>>>>>>>>> device. Ignore the bond and pretend it is a straight ethX
>>>>>>>>>>>>>>>>>> device:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> https://alteeve.ca/w/AN!Cluster_Tutorial_2#Configuring_our_Bridge.2C_Bonds_and_Interfaces
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> The host has a device named virbr0 that is installed
>>>>>>>>>>>>>>>>> during
>>>>>>>>>>>>>>>>> system
>>>>>>>>>>>>>>>>> installation. It also has a network device vnet0. There
>>>>>>>>>>>>>>>>> are no
>>>>>>>>>>>>>>>>> files in
>>>>>>>>>>>>>>>>> /etc/sysconfig/network-scripts for these.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Shouldn't I be able to use the virbr0 virtual bridge for
>>>>>>>>>>>>>>>>> this?
>>>>>>>>>>>>>>>>> I've
>>>>>>>>>>>>>>>>> tried setting up the VM's device with all of the options
>>>>>>>>>>>>>>>>> that is
>>>>>>>>>>>>>>>>> listed,
>>>>>>>>>>>>>>>>> but to no avail.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Should I need to set up another bridge for this?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> And thanks for the link.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> steve
>>>>>>>>>>>>>>>> virbr0 is created and managed by libvirtd. If you open
>>>>>>>>>>>>>>>> Virtual
>>>>>>>>>>>>>>>> Machine
>>>>>>>>>>>>>>>> Manager, connect to localhost and then double-click on
>>>>>>>>>>>>>>>> 'localhost',
>>>>>>>>>>>>>>>> you will see a tab for creating/managing bridges (NAT'ed,
>>>>>>>>>>>>>>>> generally).
>>>>>>>>>>>>>>>> I disable 'virbr0' as NAT'ing is generally not what I want.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The 'vnetX' devices are dynamically created to link a VM's
>>>>>>>>>>>>>>>> interface
>>>>>>>>>>>>>>>> to a bridge. Think of them as virtual network cables. They
>>>>>>>>>>>>>>>> get
>>>>>>>>>>>>>>>> created
>>>>>>>>>>>>>>>> and destroyed as needed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Sorry, but I'm confused:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My host server has a real NIC and IP address with a real
>>>>>>>>>>>>>>> gateway to
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> outside:
>>>>>>>>>>>>>>> virtbr0 IP: 192.168.122.1
>>>>>>>>>>>>>>> Host IP: 10.0.5.16
>>>>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>>>>> on eth0 and this works
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> My VM server has all fake stuff currently:
>>>>>>>>>>>>>>> Host IP: 10.0.5.17
>>>>>>>>>>>>>>> Gateway IP: 10.0.5.1
>>>>>>>>>>>>>>> on eth0 and this is like NIC without a cable.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> So I need to create a bridge device on both the host and VM
>>>>>>>>>>>>>>> (lets
>>>>>>>>>>>>>>> say I
>>>>>>>>>>>>>>> name it br1). I change the eth0 config file on both host and
>>>>>>>>>>>>>>> VM to
>>>>>>>>>>>>>>> point
>>>>>>>>>>>>>>> to br1 and give the br1 config file on both host and VM the
>>>>>>>>>>>>>>> correct
>>>>>>>>>>>>>>> IP.
>>>>>>>>>>>>>>> But won't this just let the two talk to each other. How will
>>>>>>>>>>>>>>> the VM
>>>>>>>>>>>>>>> server get outside?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> steve
>>>>>>>>>>>>>> The bridge is created only on the host. Think of the
>>>>>>>>>>>>>> bridge as
>>>>>>>>>>>>>> being
>>>>>>>>>>>>>> like a virtual switch. When a VM is created, you tell it to
>>>>>>>>>>>>>> connect to
>>>>>>>>>>>>>> the bridge, similar to how you would plug a physical wire
>>>>>>>>>>>>>> into a
>>>>>>>>>>>>>> real
>>>>>>>>>>>>>> switch. That provides the link to the network, and then you
>>>>>>>>>>>>>> configure
>>>>>>>>>>>>>> the virtual server's network just as if it was a real
>>>>>>>>>>>>>> network.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On the host, you don't set the IP on the ethX device,
>>>>>>>>>>>>>> instead you
>>>>>>>>>>>>>> tell
>>>>>>>>>>>>>> ethX to connect to the bridge (look at 'bond2' in the link
>>>>>>>>>>>>>> above
>>>>>>>>>>>>>> for
>>>>>>>>>>>>>> the 'BRIDGE="..."' line). Then you move the host's IP
>>>>>>>>>>>>>> address/config
>>>>>>>>>>>>>> to the bridge itself (look at the 'vbr2' in the link above).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> If you're still stuck, tell me the IP addresses you want to
>>>>>>>>>>>>>> set in
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>> host and VM and I'll see if I can bang out a couple sample
>>>>>>>>>>>>>> 'ifcfg-X'
>>>>>>>>>>>>>> files.
>>>>>>>>>>>>>>
>>>>>>>>>>>>> I'm running into some problems with my config files where I
>>>>>>>>>>>>> get a
>>>>>>>>>>>>> message that says the network can't determine my interface
>>>>>>>>>>>>> names.
>>>>>>>>>>>>> I'll
>>>>>>>>>>>>> look further, but the IPs above are what I need. 10.0.5.1 is a
>>>>>>>>>>>>> firewall
>>>>>>>>>>>>> that NATs the 10.0.5.0/24 network to another outside public
>>>>>>>>>>>>> firewall.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> steve
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> CentOS-virt mailing list
>>>>>>>>>>>>> CentOS-virt at centos.org
>>>>>>>>>>>>> http://lists.centos.org/mailman/listinfo/centos-virt
>>>>>>>>>>>>>
>>>>>>>>>>> Stuck in a rut now. Restarting the network indicates :
>>>>>>>>>>>
>>>>>>>>>>> Device eth0 does not seem to be present, delaying
>>>>>>>>>>> initialization.
>>>>>>>>>>>
>>>>>>>>>>> I thought it might be due to the fact I didn't have the
>>>>>>>>>>> "TYPE" in
>>>>>>>>>>> the
>>>>>>>>>>> ifcfg-eth0 file, but after adding that, I still get the error.
>>>>>>>>>> NM_CONTROLLED=no might help with that.
>>>>>>>>> I turned off and stopped NetworkManager and had already specified
>>>>>>>>> this.
>>>>>>>>>>> I'm using the same hardware address that was in the original
>>>>>>>>>>> eth0
>>>>>>>>>>> file.
>>>>>>>>>> That is fine.
>>>>>>>>>>
>>>>>>>>>>>        My bridge device starts fine, but I'll tailor it to your
>>>>>>>>>>> specs.
>>>>>>>>>> Also, you might want to check the contents of your
>>>>>>>>>> /etc/udev/rules.d/70-persistent-net.rules file. Is the MAC
>>>>>>>>>> address
>>>>>>>>>> for
>>>>>>>>>> eth0 the same as in your
>>>>>>>>>> /etc/sysconfig/network-scripts/ifcfg-eth0
>>>>>>>>>> file?
>>>>>>>>>>
>>>>>>>>> They are the same except the udev file has lower case letters
>>>>>>>>> in it.
>>>>>>>> You mean the hexadecimal numbers are lower case? That shouldn't
>>>>>>>> matter.
>>>>>>>>
>>>>>>>> Try to delete the file /etc/udev/rules.d/70-persistent-net.rules
>>>>>>>> and
>>>>>>>> reboot your host machine.
>>>>>>>> ( See
>>>>>>>> http://www.andrewklau.com/device-eth0-does-not-seem-to-be-present-delaying-initialization/
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> )
>>>>>>>>
>>>>>>> All is well on the host now, and I've brought up the VM and
>>>>>>> configured
>>>>>>> it to use the bridge. I can ping the host from the VM but I have no
>>>>>>> access beyond the local network. The host sees the world, but the VM
>>>>>>> only sees the 10.0.5.0/24 network.
>>>>>> Did you set the gateway in your VM to 10.0.5.1 ?
>>>>>> Can you ping the gateway ( ping 10.0.5.1 ) from the VM?
>>>>>> Do you have a firewall configured in your VM (or host) ? If so,
>>>>>> try to
>>>>>> disable the firewall temporarily for testing.
>>>>>>
>>>>> The gateway is set. Can't ping the gateway. Firewalls (iptables and
>>>>> ip6tables) are stopped and off. Previous post suggested the route.
>>>>> I'll
>>>>> check it.
>>>> Add the following lines to your /etc/sysctl.conf file on your host:
>>>>
>>>> net.ipv4.ip_forward = 1
>>>> net.ipv4.conf.default.rp_filter = 0
>>>> net.ipv6.conf.all.disable_ipv6 = 1
>>>> net.ipv6.conf.default.disable_ipv6 = 1
>>>>
>>>> then issue the following command:
>>>>
>>>> sysctl -p
>>>>
>>> Still no joy. I'm installing another VM just to see if having the bridge
>>> defined before installation would make a difference. I find it hard to
>>> imagine all of this can't be done will creating the VM (Xen seemed to do
>>> it all for you).
>> What you did on the host machine, only has to be done once when you
>> setup the system to be used as a KVM host server.
>>
>>> So now, I've got the route, gateway, bridge, and eth0/1 defined on the
>>> original. I can ping both the host and the VM from each other. The host
>>> can see the world through the gateway. The VM can't see the gateway or
>>> the world.
>> How did you come up with the IP address ( 10.0.5.17 ) for the VM ? Are
>> you sure it is not blocked somehow by your gateway ( at 10.0.5.1 ) ?
>>
>> Also, maybe your gateway ( at 10.0.5.1 ) is doing MAC address
>> filtering? Since your VM has its own MAC address, this could be a
>> problem.
>>
> The gateway is a firewall. It's set up to route and NAT from 10.0.5.0/24
> to another inside network on 10.0.0.0/24. This is the true inside
> network that is NATted to the public with our public IPs. It accepts all
> traffic from the 10.0.5.0 network.
>
> steve
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt
>
I created another VM, this time the bridge was available when it was 
created, an it's working. I'll delete the first VM and re-create it and 
see if that doesn't resolve it.

Thanks for all the help and tutoring on this.

steve