I have decided to migrate my latest KVM server to CentOS 6.0 and am beginning to get a little frustrated with some issues that worked perfectly in 5.6.
Right now I've given up on getting virbr0 and NAT to work, but now I need networking bridging to work, but nothing seems to fix the issue. I have not had much experience with troubleshooting KVM so could really use some pointers on resolving this issue.
I have successfully bridged one of the server's NICs to br0, and I can ping the IP remotely that is assigned to br0, but none of the VMs that worked in 5.6's KVM are able to access the network. Please let me know what information would be useful to troubleshoot this.
Here's what I have so far...
ifcfg-eth3... --------------------------------- DEVICE="eth3" HWADDR="00:1B:21:A1:CF:76" ONBOOT="yes" BRIDGE=br0
ifcfg-br0 -------------------------------- DEVICE=br0 TYPE=Bridge BOOTPROTO=static ONBOOT=yes IPADDR=.... BROADCAST=.... NETMASK=.... NETWORK=....
# brctl show bridge name bridge id STP enabled interfaces br0 8000.001b21a1cf76 no eth3
sysctl.conf has these 3 additional lines... net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0
Again I can ping br0's IP remotely , but no VMs assigned to it can access even that network's gateway. I have disabled iptables and ip6tables for now while trying to get this to work. The VMs , worked just fine in 5.6 and I have done nothing different but import them with virsh.
Thanks - Trey
No experience with 6 here, but do your virsh-imported libvirt VM configs show <devices>...<interface type=bridge>...<source bridge='br0'/> ? i.e. the bridge there matches the bridge name you're created with ifcfg scripts?
Eric
On Fri, Jul 15, 2011 at 4:20 PM, Eric Searcy emsearcy@gmail.com wrote:
No experience with 6 here, but do your virsh-imported libvirt VM configs show <devices>...<interface type=bridge>...<source bridge='br0'/> ? i.e. the bridge there matches the bridge name you're created with ifcfg scripts?
Eric _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Yes, the name matches, the KVM hosts both used br0 on 5.6 and now on 6.0. Here's what's in the XML file
<interface type='bridge'> <mac address='52:54:00:31:6f:16'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface>
Thanks - Trey
On 16.7.2011 2.15, Trey Dockendorf wrote:
Yes, the name matches, the KVM hosts both used br0 on 5.6 and now on 6.0. Here's what's in the XML file
<interface type='bridge'> <mac address='52:54:00:31:6f:16'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'function='0x0'/> </interface>
I am not sure if yo have the same kind of virtual network structure as I have. Anyway, my xml file (CentOS 5.6 on both host and guest) says simply: <interface type='bridge'> <mac address='54:52:00:3d:aa:46'/> <source bridge='br0'/> <model type='virtio'/> </interface>
I have one physical nic, one bridge (br0), one external ip address per guest. No NAT.
BTW, I now notice that I have wrong MAC address (54:52), and you have it right (52:54), I probably got mine from "man virt-install" which contains an error:
https://bugzilla.redhat.com/show_bug.cgi?id=620838
Anyway, my vguests work fine.
- Jussi
On Fri, Jul 15, 2011 at 2:07 PM, Trey Dockendorf treydock@gmail.com wrote:
I have decided to migrate my latest KVM server to CentOS 6.0 and am beginning to get a little frustrated with some issues that worked perfectly in 5.6.
(snip)
Here's what I have so far... ifcfg-eth3...
DEVICE="eth3" HWADDR="00:1B:21:A1:CF:76" ONBOOT="yes" BRIDGE=br0 ifcfg-br0
DEVICE=br0 TYPE=Bridge BOOTPROTO=static ONBOOT=yes IPADDR=.... BROADCAST=.... NETMASK=.... NETWORK=....
I do not see GATEWAY=.... in your ifcfg-br0. Do you actually have it?
Again I can ping br0's IP remotely , but no VMs assigned to it can access even that network's gateway.
Akemi
On Fri, Jul 15, 2011 at 4:20 PM, Akemi Yagi amyagi@gmail.com wrote:
On Fri, Jul 15, 2011 at 2:07 PM, Trey Dockendorf treydock@gmail.com wrote:
I have decided to migrate my latest KVM server to CentOS 6.0 and am beginning to get a little frustrated with some issues that worked
perfectly
in 5.6.
(snip)
Here's what I have so far... ifcfg-eth3...
DEVICE="eth3" HWADDR="00:1B:21:A1:CF:76" ONBOOT="yes" BRIDGE=br0 ifcfg-br0
DEVICE=br0 TYPE=Bridge BOOTPROTO=static ONBOOT=yes IPADDR=.... BROADCAST=.... NETMASK=.... NETWORK=....
I do not see GATEWAY=.... in your ifcfg-br0. Do you actually have it?
Again I can ping br0's IP remotely , but no VMs assigned to it can access even that network's gateway.
Akemi _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
The GATEWAY is specified in /etc/sysconfig/network file. Should I include it in the ifcfg-br0 as well? In CentOS 5.x I always left it in the /etc/sysconfig/network file which seems to be the default from fresh install of CentOS 6.
Thanks - Trey
On 7/16/11, Trey Dockendorf treydock@gmail.com wrote:
I have successfully bridged one of the server's NICs to br0, and I can ping the IP remotely that is assigned to br0, but none of the VMs that worked in 5.6's KVM are able to access the network. Please let me know what information would be useful to troubleshoot this.
Could you try creating a new VM using the GUI tool, then check if the networking works from it?
I was having problems with KVM and part of the troubleshooting process got me to try it on SL6, which finally led me to discover that the command line tool generated XML doesn't work as well as the GUI tool for some reason. So there's the possibility that it could be that the definitions created through virsh in 5.6 has the same issues in CentOS 6.
On Fri, Jul 15, 2011 at 6:24 PM, Emmanuel Noobadmin centos.admin@gmail.comwrote:
On 7/16/11, Trey Dockendorf treydock@gmail.com wrote:
I have successfully bridged one of the server's NICs to br0, and I can
ping
the IP remotely that is assigned to br0, but none of the VMs that worked
in
5.6's KVM are able to access the network. Please let me know what information would be useful to troubleshoot this.
Could you try creating a new VM using the GUI tool, then check if the networking works from it?
I was having problems with KVM and part of the troubleshooting process got me to try it on SL6, which finally led me to discover that the command line tool generated XML doesn't work as well as the GUI tool for some reason. So there's the possibility that it could be that the definitions created through virsh in 5.6 has the same issues in CentOS 6. _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
I did try another VM (CentOS 6) via virt-manager with the same results. However I setup a test server at home, and am able to get both bridging and NAT to work so this may be an issue with the network on my server. It's a University network and their switches tend to play havoc with virtual servers even though I've been assured enough MAC addresses have been allowed on my port.
How does one troubleshoot or provide debug information on a correctly or incorrectly functioning network bridge? As I contact my University's helpdesk I'd like to be able to point out the fault is not with my KVM server.
Thanks - Trey
When you showed the output of brctl show earlier only eth0 showed up. Does the VM NIC showed up attached to br0 when the VM is running?
If so, then you can ping the VM from your host? That wouldn't involve the university switches so it would be a good first step before digging into packet dumping arp traffic...
Sent from my mobile phone
On Jul 16, 2011, at 4:58 PM, Trey Dockendorf treydock@gmail.com wrote:
On Fri, Jul 15, 2011 at 6:24 PM, Emmanuel Noobadmin centos.admin@gmail.com wrote: On 7/16/11, Trey Dockendorf treydock@gmail.com wrote:
I have successfully bridged one of the server's NICs to br0, and I can ping the IP remotely that is assigned to br0, but none of the VMs that worked in 5.6's KVM are able to access the network. Please let me know what information would be useful to troubleshoot this.
Could you try creating a new VM using the GUI tool, then check if the networking works from it?
I was having problems with KVM and part of the troubleshooting process got me to try it on SL6, which finally led me to discover that the command line tool generated XML doesn't work as well as the GUI tool for some reason. So there's the possibility that it could be that the definitions created through virsh in 5.6 has the same issues in CentOS 6. _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
I did try another VM (CentOS 6) via virt-manager with the same results. However I setup a test server at home, and am able to get both bridging and NAT to work so this may be an issue with the network on my server. It's a University network and their switches tend to play havoc with virtual servers even though I've been assured enough MAC addresses have been allowed on my port.
How does one troubleshoot or provide debug information on a correctly or incorrectly functioning network bridge? As I contact my University's helpdesk I'd like to be able to point out the fault is not with my KVM server.
Thanks
- Trey
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
On 07/16/2011 04:58 PM, Trey Dockendorf wrote:
On Fri, Jul 15, 2011 at 6:24 PM, Emmanuel Noobadmin <centos.admin@gmail.com mailto:centos.admin@gmail.com> wrote:
On 7/16/11, Trey Dockendorf <treydock@gmail.com <mailto:treydock@gmail.com>> wrote: > I have successfully bridged one of the server's NICs to br0, and I can ping > the IP remotely that is assigned to br0, but none of the VMs that worked in > 5.6's KVM are able to access the network. Please let me know what > information would be useful to troubleshoot this. Could you try creating a new VM using the GUI tool, then check if the networking works from it? I was having problems with KVM and part of the troubleshooting process got me to try it on SL6, which finally led me to discover that the command line tool generated XML doesn't work as well as the GUI tool for some reason. So there's the possibility that it could be that the definitions created through virsh in 5.6 has the same issues in CentOS 6. _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org <mailto:CentOS-virt@centos.org> http://lists.centos.org/mailman/listinfo/centos-virtI did try another VM (CentOS 6) via virt-manager with the same results. However I setup a test server at home, and am able to get both bridging and NAT to work so this may be an issue with the network on my server. It's a University network and their switches tend to play havoc with virtual servers even though I've been assured enough MAC addresses have been allowed on my port.
How does one troubleshoot or provide debug information on a correctly or incorrectly functioning network bridge? As I contact my University's helpdesk I'd like to be able to point out the fault is not with my KVM server.
Thanks
- Trey
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
If both the VM's and the server are on the same bridge and they can't talk to each other, I would from both the server and VM end, ping the opposite end and check the arp table to see if arp entries are getting resolved, then I would run tcpdump on each respective end and send packets from the other end and see if they are getting through. If not, then their is either a problem with either the VM's config file or the networking/bridge config on the server. (Of course if you have any kind of ipfilter access lists, then I would check those).
Once you've got the above working, I would attempt to perform similar tests to the outside. If you happen to have a login on another host on the same subnet, you can ping your VM and check the arp table to see if there is arp resolution. (Also check that you don't have duplicate ip address assignments). If there is arp resolution, then run tcp dump on the vm (or the physical interface of the server) and see if you can see the packets from outside. Checking the reverse direction is harder if you don't have root access on the remote end.
If your going through gateways, then run traceroute to see how far your getting.
As I thought about it more, it's unclear weather your VM's are on a seperate bridge from your server's external interface or they are bridged directly onto it. If the VM's are on a bridge that also has an external interface on it, then you don't use NAT. NAT would be if you wanted your server to act as a router/firewall for the VM's in which case the VM's would be on a separate bridge and the server would have another external interface and would act as a router between the bridge network and the external network.
This should be a start anyway.
Nataraj
Nataraj , I'm going to work on your suggestions, but I have a bit more troubleshooting info that may help (or to me just makes this stranger), and also to answer Eric's suggestion.
With the VM running this is the brctl table...
# brctl show bridge name bridge id STP enabled interfaces br0 8000.001b21a1cf76 no eth3 vnet0 virbr0 8000.fe5400345b8c yes vnet1
So that looks good.
The arp table from the VM only shows its own interface, nothing else.
From the KVM host, I get this (with IPs removed)...
# arp Address HWtype HWaddress Flags Mask Iface 192.168.122.20 ether 52:54:00:34:5b:8c C virbr0 ---.---.---.--- (VM on br0) (incomplete) eth0 ---.---.---.--- (gateway) ether 00:13:xx:xx:xx:xx C eth0
To clarify a bit about the networking on this server, there are 4 physical interfaces (eth0-3). eth0 is what I use for management. eth3 is linked to br0 and that is the bridge the VM is associated to. The VM currently has two interfaces (for testing in this case). the VM's eth0 is attached to host's br0 and the VMs eth1 is attached to the host's NAT.
What makes this situation very confusing to me is that from the VM, with NAT now working...I can ping my servers on other subnets, and resolve DNS (ie ping google.com). However I am unable to ping the gateway of the local subnet or the KVM host, from the VM, even though networking is functional using NAT. I am also unable to ping the VM from the KVM host via the bridged IP. This is what leads me to believe the university switches could be blocking something. I've been told they increased the MAC address limit on my port, but due to restrictions on this subnet I am not able to directly correspond or troubleshoot this issue with the University IT , but rather have to go through someone else who only knows Windows (*sigh*).
I'll work on troubleshooting with tcpdump and traceroute, but have never used those tools previously so will have to do a bit of quick learning-on-the-go to provide useful information.
Thanks for the help thus far!
- Trey
On Sat, Jul 16, 2011 at 8:24 PM, Nataraj incoming-centos@rjl.com wrote:
** On 07/16/2011 04:58 PM, Trey Dockendorf wrote:
On Fri, Jul 15, 2011 at 6:24 PM, Emmanuel Noobadmin < centos.admin@gmail.com> wrote:
On 7/16/11, Trey Dockendorf treydock@gmail.com wrote:
I have successfully bridged one of the server's NICs to br0, and I can
ping
the IP remotely that is assigned to br0, but none of the VMs that worked
in
5.6's KVM are able to access the network. Please let me know what information would be useful to troubleshoot this.
Could you try creating a new VM using the GUI tool, then check if the networking works from it?
I was having problems with KVM and part of the troubleshooting process got me to try it on SL6, which finally led me to discover that the command line tool generated XML doesn't work as well as the GUI tool for some reason. So there's the possibility that it could be that the definitions created through virsh in 5.6 has the same issues in CentOS 6. _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
I did try another VM (CentOS 6) via virt-manager with the same results. However I setup a test server at home, and am able to get both bridging and NAT to work so this may be an issue with the network on my server. It's a University network and their switches tend to play havoc with virtual servers even though I've been assured enough MAC addresses have been allowed on my port.
How does one troubleshoot or provide debug information on a correctly or incorrectly functioning network bridge? As I contact my University's helpdesk I'd like to be able to point out the fault is not with my KVM server.
Thanks
- Trey
CentOS-virt mailing listCentOS-virt@centos.orghttp://lists.centos.org/mailman/listinfo/centos-virt
If both the VM's and the server are on the same bridge and they can't talk to each other, I would from both the server and VM end, ping the opposite end and check the arp table to see if arp entries are getting resolved, then I would run tcpdump on each respective end and send packets from the other end and see if they are getting through. If not, then their is either a problem with either the VM's config file or the networking/bridge config on the server. (Of course if you have any kind of ipfilter access lists, then I would check those).
Once you've got the above working, I would attempt to perform similar tests to the outside. If you happen to have a login on another host on the same subnet, you can ping your VM and check the arp table to see if there is arp resolution. (Also check that you don't have duplicate ip address assignments). If there is arp resolution, then run tcp dump on the vm (or the physical interface of the server) and see if you can see the packets from outside. Checking the reverse direction is harder if you don't have root access on the remote end.
If your going through gateways, then run traceroute to see how far your getting.
As I thought about it more, it's unclear weather your VM's are on a seperate bridge from your server's external interface or they are bridged directly onto it. If the VM's are on a bridge that also has an external interface on it, then you don't use NAT. NAT would be if you wanted your server to act as a router/firewall for the VM's in which case the VM's would be on a separate bridge and the server would have another external interface and would act as a router between the bridge network and the external network.
This should be a start anyway.
Nataraj
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
On Sat, Jul 16, 2011 at 11:22 PM, Trey Dockendorf treydock@gmail.comwrote:
Nataraj , I'm going to work on your suggestions, but I have a bit more troubleshooting info that may help (or to me just makes this stranger), and also to answer Eric's suggestion.
With the VM running this is the brctl table...
# brctl show bridge name bridge id STP enabled interfaces br0 8000.001b21a1cf76 no eth3 vnet0 virbr0 8000.fe5400345b8c yes vnet1
So that looks good.
The arp table from the VM only shows its own interface, nothing else.
From the KVM host, I get this (with IPs removed)...
# arp Address HWtype HWaddress Flags Mask Iface 192.168.122.20 ether 52:54:00:34:5b:8c C virbr0 ---.---.---.--- (VM on br0) (incomplete) eth0 ---.---.---.--- (gateway) ether 00:13:xx:xx:xx:xx C eth0
To clarify a bit about the networking on this server, there are 4 physical interfaces (eth0-3). eth0 is what I use for management. eth3 is linked to br0 and that is the bridge the VM is associated to. The VM currently has two interfaces (for testing in this case). the VM's eth0 is attached to host's br0 and the VMs eth1 is attached to the host's NAT.
What makes this situation very confusing to me is that from the VM, with NAT now working...I can ping my servers on other subnets, and resolve DNS (ie ping google.com). However I am unable to ping the gateway of the local subnet or the KVM host, from the VM, even though networking is functional using NAT. I am also unable to ping the VM from the KVM host via the bridged IP. This is what leads me to believe the university switches could be blocking something. I've been told they increased the MAC address limit on my port, but due to restrictions on this subnet I am not able to directly correspond or troubleshoot this issue with the University IT , but rather have to go through someone else who only knows Windows (*sigh*).
I'll work on troubleshooting with tcpdump and traceroute, but have never used those tools previously so will have to do a bit of quick learning-on-the-go to provide useful information.
Thanks for the help thus far!
- Trey
On Sat, Jul 16, 2011 at 8:24 PM, Nataraj incoming-centos@rjl.com wrote:
** On 07/16/2011 04:58 PM, Trey Dockendorf wrote:
On Fri, Jul 15, 2011 at 6:24 PM, Emmanuel Noobadmin < centos.admin@gmail.com> wrote:
On 7/16/11, Trey Dockendorf treydock@gmail.com wrote:
I have successfully bridged one of the server's NICs to br0, and I can
ping
the IP remotely that is assigned to br0, but none of the VMs that
worked in
5.6's KVM are able to access the network. Please let me know what information would be useful to troubleshoot this.
Could you try creating a new VM using the GUI tool, then check if the networking works from it?
I was having problems with KVM and part of the troubleshooting process got me to try it on SL6, which finally led me to discover that the command line tool generated XML doesn't work as well as the GUI tool for some reason. So there's the possibility that it could be that the definitions created through virsh in 5.6 has the same issues in CentOS 6. _______________________________________________ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
I did try another VM (CentOS 6) via virt-manager with the same results. However I setup a test server at home, and am able to get both bridging and NAT to work so this may be an issue with the network on my server. It's a University network and their switches tend to play havoc with virtual servers even though I've been assured enough MAC addresses have been allowed on my port.
How does one troubleshoot or provide debug information on a correctly or incorrectly functioning network bridge? As I contact my University's helpdesk I'd like to be able to point out the fault is not with my KVM server.
Thanks
- Trey
CentOS-virt mailing listCentOS-virt@centos.orghttp://lists.centos.org/mailman/listinfo/centos-virt
If both the VM's and the server are on the same bridge and they can't talk to each other, I would from both the server and VM end, ping the opposite end and check the arp table to see if arp entries are getting resolved, then I would run tcpdump on each respective end and send packets from the other end and see if they are getting through. If not, then their is either a problem with either the VM's config file or the networking/bridge config on the server. (Of course if you have any kind of ipfilter access lists, then I would check those).
Once you've got the above working, I would attempt to perform similar tests to the outside. If you happen to have a login on another host on the same subnet, you can ping your VM and check the arp table to see if there is arp resolution. (Also check that you don't have duplicate ip address assignments). If there is arp resolution, then run tcp dump on the vm (or the physical interface of the server) and see if you can see the packets from outside. Checking the reverse direction is harder if you don't have root access on the remote end.
If your going through gateways, then run traceroute to see how far your getting.
As I thought about it more, it's unclear weather your VM's are on a seperate bridge from your server's external interface or they are bridged directly onto it. If the VM's are on a bridge that also has an external interface on it, then you don't use NAT. NAT would be if you wanted your server to act as a router/firewall for the VM's in which case the VM's would be on a separate bridge and the server would have another external interface and would act as a router between the bridge network and the external network.
This should be a start anyway.
Nataraj
CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Apologies for replying to my own post, but after running arp a few more times on the VM it now shows the MAC and IP of the subnet's gateway and the KVM host. Still not entirely sure what all this information means, but it's a start.
Thanks
- Trey
-
Akemi Yagi -
Emmanuel Noobadmin -
Eric Searcy -
Jussi Hirvi -
Nataraj -
Trey Dockendorf