Ran into a strange issue with XEN on CentOS that I think is specific to CentOS, which is why I'm starting by posting to this list first, I'll post on the XEN list depending on responses. My sense is this issue has something to do with how CentOS handles network setup on first boot of the XEN kernel.
- Installed a brand new CentOS 5.3 server with minimal packages.
- Installed XEN, modified grub.conf to boot off of the XEN kernel and rebooted.
- After reboot, network connectivity was lost.
- Investigation concluded the issue was that the HWaddr address of the physical NIC matched the fabricated HWaddr that XEN uses for most of its adapters: FE:FF:FF:FF:FF:FF.
- Temporary resolution, I re-enabled the motherboard's NIC, rebooted, all seems to be working.
I would like to get the NIC in question working as it is a GigaBit NIC, but it still has the HWaddr: FE:FF:FF:FF:FF:FF which conflicts with XEN.
My understanding of the HWaddr is that the first portion is manufacturer assigned for uniqueness, I cannot image this NIC originally had this HWaddr, but I don't know what it originally was.
Does anyone know if this value is read from the NIC on each boot, or is it stored in a file after the first boot? Is there someway to undo this change so the NIC returns to its original value or atleast a non-conflicting value?
Has anyone else seen this behavior?
Thank you in advance,
Brett
Am Dienstag, den 04.08.2009, 14:43 +0200 schrieb Brett Serkez:
Ran into a strange issue with XEN on CentOS that I think is specific to CentOS, which is why I'm starting by posting to this list first, I'll post on the XEN list depending on responses. My sense is this issue has something to do with how CentOS handles network setup on first boot of the XEN kernel.
Installed a brand new CentOS 5.3 server with minimal packages.
Installed XEN, modified grub.conf to boot off of the XEN kernel and rebooted.
After reboot, network connectivity was lost.
Investigation concluded the issue was that the HWaddr address of the
physical NIC matched the fabricated HWaddr that XEN uses for most of its adapters: FE:FF:FF:FF:FF:FF.
When Xen starts does some trickery with your interfaces. You should see FE:FF:FF:FF:FF:FF on device peth0 and the real MAC-address on device eth0. All Xen vif devices will show also MAC FE:FF:FF:FF:FF:FF. That is totally normal.
Chris
financial.com AG
Munich head office/Hauptsitz München: Maria-Probst-Str. 19 | 80939 München | Germany Frankfurt branch office/Niederlassung Frankfurt: Messeturm | Friedrich-Ebert-Anlage 49 | 60327 Frankfurt | Germany Management board/Vorstand: Dr. Steffen Boehnert (CEO/Vorsitzender) | Dr. Alexis Eisenhofer | Dr. Yann Samson | Matthias Wiederwach Supervisory board/Aufsichtsrat: Dr. Dr. Ernst zur Linden (chairman/Vorsitzender) Register court/Handelsregister: Munich – HRB 128 972 | Sales tax ID number/St.Nr.: DE205 370 553
On Tue, Aug 4, 2009 at 9:09 AM, Christoph Masercmr@financial.com wrote: <snip>
When Xen starts does some trickery with your interfaces. You should see FE:FF:FF:FF:FF:FF on device peth0 and the real MAC-address on device eth0. All Xen vif devices will show also MAC FE:FF:FF:FF:FF:FF. That is totally normal.
I wanted to clarify on this point. Understood as far as the above, but the issue is that the PHYSICAL MAC was changed to FE:FF:FF:FF:FF:FF.
This went so far as to still have this value even after rebooting on the standard kernel and then uninstalling XEN:
# dmesg | grep eth1 eth1: RTL8110s at 0xee156c00, fe:ff:ff:ff:ff:ff, XID 04000000 IRQ 22
What ever happened during the first boot of XEN caused a permanent change to this NIC as far as I can tell.
Brett
On Tue, Aug 4, 2009 at 12:30 PM, Brett Serkezbserkez@gmail.com wrote:
On Tue, Aug 4, 2009 at 9:09 AM, Christoph Masercmr@financial.com wrote:
<snip> > When Xen starts does some trickery with your interfaces. You should see > FE:FF:FF:FF:FF:FF on device peth0 and the real MAC-address on device > eth0. > All Xen vif devices will show also MAC FE:FF:FF:FF:FF:FF. That is > totally normal.
I wanted to clarify on this point. Understood as far as the above, but the issue is that the PHYSICAL MAC was changed to FE:FF:FF:FF:FF:FF.
This went so far as to still have this value even after rebooting on the standard kernel and then uninstalling XEN:
# dmesg | grep eth1 eth1: RTL8110s at 0xee156c00, fe:ff:ff:ff:ff:ff, XID 04000000 IRQ 22
What ever happened during the first boot of XEN caused a permanent change to this NIC as far as I can tell.
Maybe because you are looking at the bridge's mac and not the ethernet's which would be peth0.
-Ross
On Aug 4, 2009, at 12:50 PM, Ross Walker rswwalker@gmail.com wrote:
On Tue, Aug 4, 2009 at 12:30 PM, Brett Serkezbserkez@gmail.com wrote:
On Tue, Aug 4, 2009 at 9:09 AM, Christoph Masercmr@financial.com wrote:
<snip> > When Xen starts does some trickery with your interfaces. You > should see > FE:FF:FF:FF:FF:FF on device peth0 and the real MAC-address on device > eth0. > All Xen vif devices will show also MAC FE:FF:FF:FF:FF:FF. That is > totally normal.
I wanted to clarify on this point. Understood as far as the above, but the issue is that the PHYSICAL MAC was changed to FE:FF:FF:FF:FF:FF.
This went so far as to still have this value even after rebooting on the standard kernel and then uninstalling XEN:
# dmesg | grep eth1 eth1: RTL8110s at 0xee156c00, fe:ff:ff:ff:ff:ff, XID 04000000 IRQ 22
What ever happened during the first boot of XEN caused a permanent change to this NIC as far as I can tell.
Maybe because you are looking at the bridge's mac and not the ethernet's which would be peth0.
Actually that would be peth1
For a Xen server I find it better to disable the Xen network scripts and setup a static bridged or routed setup.
Then it is predictable and iptables is infinitely easier to setup.
-Ross
Maybe because you are looking at the bridge's mac and not the ethernet's which would be peth0.
No I am not. dmesg shows the kernel messages at boot and it is looking at the physical device, let's not get distracted, the issue is clear in this regard. As I previously stated, this happens even when uninstalling XEN and booting off the non-XEN kernel since the install of XEN.
indeed, AFAIK all hardware adapters start with 00. This must have been set in the BIOS or with a boot option or in the network config.
This was helpful, gave me places/incentive to continue looking.
In /etc/sysconfig/network-scripts/ifcfg-eth1 I found:
# Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet DEVICE=eth1 BOOTPROTO=none HWADDR=00:40:F4:CE:E6:7B
So now I know what the original MAC address was.
Here is where it gets interesting. The following file was modified at the date/time that the XEN kernel was first booted:
/etc/sysconfig/hwconf
and it has fe:ff:ff:ff:ff:ff for BOTH network adapters:
desc: "Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet" networfe:ffddr: fe:ff:ff:ff:ff:ff vendorId: 10ec deviceId: 8169 subVendorId: 10ec subDeviceId: 8169 pciType: 10
desc: "Intel Corporation 82562EZ 10/100 Ethernet Controller" network.hwaddr: fe:ff:ff:ff:ff:ff vendorId: 8086 deviceId: 1050 subVendorId: 8086 subDeviceId: 303a
Everything I'm finding is re-enforcing my original theory that XEN modified the hwaddr of this NIC.
The question continues to be what caused this and how to change it back. Given this is a stock system, I have to believe others must have/may run into this issue.
Brett
Brett,
I think the following link answers your question about the MAC changes. You may find more useful links on the resources page of the Running Xen site http://runningxen.com/resources/.
http://lists.xensource.com/archives/html/xen-users/2006-02/msg00030.html
If you performed a fresh install without Xen, you would notice that it has not permanently modified the MAC address of your system.
Hope that helps.
Matt
-- Mathew S. McCarrell Clarkson University '10
mccarrms@gmail.com mccarrms@clarkson.edu 1-518-314-9214
On Tue, Aug 4, 2009 at 1:30 PM, Brett Serkez bserkez@gmail.com wrote:
Maybe because you are looking at the bridge's mac and not the ethernet's which would be peth0.
No I am not. dmesg shows the kernel messages at boot and it is looking at the physical device, let's not get distracted, the issue is clear in this regard. As I previously stated, this happens even when uninstalling XEN and booting off the non-XEN kernel since the install of XEN.
indeed, AFAIK all hardware adapters start with 00. This must have been
set
in the BIOS or with a boot option or in the network config.
This was helpful, gave me places/incentive to continue looking.
In /etc/sysconfig/network-scripts/ifcfg-eth1 I found:
# Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet DEVICE=eth1 BOOTPROTO=none HWADDR=00:40:F4:CE:E6:7B
So now I know what the original MAC address was.
Here is where it gets interesting. The following file was modified at the date/time that the XEN kernel was first booted:
/etc/sysconfig/hwconf
and it has fe:ff:ff:ff:ff:ff for BOTH network adapters:
desc: "Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet" networfe:ffddr: fe:ff:ff:ff:ff:ff vendorId: 10ec deviceId: 8169 subVendorId: 10ec subDeviceId: 8169 pciType: 10
desc: "Intel Corporation 82562EZ 10/100 Ethernet Controller" network.hwaddr: fe:ff:ff:ff:ff:ff vendorId: 8086 deviceId: 1050 subVendorId: 8086 subDeviceId: 303a
Everything I'm finding is re-enforcing my original theory that XEN modified the hwaddr of this NIC.
The question continues to be what caused this and how to change it back. Given this is a stock system, I have to believe others must have/may run into this issue.
Brett _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Brett Serkez wrote:
Maybe because you are looking at the bridge's mac and not the ethernet's which would be peth0.
No I am not. dmesg shows the kernel messages at boot and it is looking at the physical device, let's not get distracted, the issue is clear in this regard. As I previously stated, this happens even when uninstalling XEN and booting off the non-XEN kernel since the install of XEN.
indeed, AFAIK all hardware adapters start with 00. This must have been set in the BIOS or with a boot option or in the network config.
This was helpful, gave me places/incentive to continue looking.
In /etc/sysconfig/network-scripts/ifcfg-eth1 I found:
# Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet DEVICE=eth1 BOOTPROTO=none HWADDR=00:40:F4:CE:E6:7B
So now I know what the original MAC address was.
Here is where it gets interesting. The following file was modified at the date/time that the XEN kernel was first booted:
/etc/sysconfig/hwconf
and it has fe:ff:ff:ff:ff:ff for BOTH network adapters:
desc: "Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet" networfe:ffddr: fe:ff:ff:ff:ff:ff vendorId: 10ec deviceId: 8169 subVendorId: 10ec subDeviceId: 8169 pciType: 10
desc: "Intel Corporation 82562EZ 10/100 Ethernet Controller" network.hwaddr: fe:ff:ff:ff:ff:ff vendorId: 8086 deviceId: 1050 subVendorId: 8086 subDeviceId: 303a
Everything I'm finding is re-enforcing my original theory that XEN modified the hwaddr of this NIC.
The question continues to be what caused this and how to change it back. Given this is a stock system, I have to believe others must have/may run into this issue.
Brett _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
There is a bug filed against the RHEL5 upstream on the Realtek (r8169) driver. I've experienced it as well. The problem is unresolved in current kernels. See https://bugzilla.redhat.com/show_bug.cgi?id=503988
Since you know what the original MAC address was, you can try: ifconfig eth0 down ifconfig eth0 hw ether 00:40:F4:CE:E6:7B ifconfig eth0 up
Note that: * You can't change the MAC address with the interface plumbed, hence the down/up sequence. * You'll lose all routes associated with the interface when it goes down, so you'll have to re-add the routes or restart the networking service to get them back * It'll happen again.
You can either try the alternate networking configs for Xen, or add some of this stuff to your boot sequence right after the xend service starts to work around it. I assume you've found, as I did, that after Xen started and the MAC reset, you lost connectivity, maybe even saw odd messages about 'packet arrived with my source address.'
Hope this helped, good luck.
On Tue, Aug 4, 2009 at 12:30 PM, Brett Serkezbserkez@gmail.com wrote: <snip>
# Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Ethernet DEVICE=eth1 BOOTPROTO=none HWADDR=00:40:F4:CE:E6:7B
So now I know what the original MAC address was.
Is it possible for the MAC address to be changed physically on the NIC? I doubt that. In SW, you can have an alternate MAC address, but I doubt that it can be changed on the NIC. That's burned into the NIC at the factory.
Hi,
This commonly happens when you using Xen in the bridged mode (when you reboot your system the first time, this is default Xen configuration). You have to change your configuration to routed mode if you want to prevent that in future.
You can get more info about network in Xen going by links below -
http://wiki.xensource.com/xenwiki/XenNetworking#head-d5446face7e308f577e5aee...
P.S. my first experience with Xen starting from the same issue like your (is was a problem for me, because I have dedicated server and support stuff from hosting company every time when the NIC address was changed want to kick me)... :)
On Tue, Aug 4, 2009 at 4:43 PM, Brett Serkezbserkez@gmail.com wrote:
Ran into a strange issue with XEN on CentOS that I think is specific to CentOS, which is why I'm starting by posting to this list first, I'll post on the XEN list depending on responses. My sense is this issue has something to do with how CentOS handles network setup on first boot of the XEN kernel.
Installed a brand new CentOS 5.3 server with minimal packages.
Installed XEN, modified grub.conf to boot off of the XEN kernel and rebooted.
After reboot, network connectivity was lost.
Investigation concluded the issue was that the HWaddr address of the
physical NIC matched the fabricated HWaddr that XEN uses for most of its adapters: FE:FF:FF:FF:FF:FF.
- Temporary resolution, I re-enabled the motherboard's NIC, rebooted,
all seems to be working.
I would like to get the NIC in question working as it is a GigaBit NIC, but it still has the HWaddr: FE:FF:FF:FF:FF:FF which conflicts with XEN.
My understanding of the HWaddr is that the first portion is manufacturer assigned for uniqueness, I cannot image this NIC originally had this HWaddr, but I don't know what it originally was.
Does anyone know if this value is read from the NIC on each boot, or is it stored in a file after the first boot? Is there someway to undo this change so the NIC returns to its original value or atleast a non-conflicting value?
Has anyone else seen this behavior?
Thank you in advance,
Brett _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Sergey Smirnov wrote on Tue, 4 Aug 2009 17:14:48 +0400:
This commonly happens when you using Xen in the bridged mode (when you reboot your system the first time, this is default Xen configuration). You have to change your configuration to routed mode if you want to prevent that in future.
No, I haven't ever seen that the physical MAC address got changed to FE- whatever by Xen. Anyway, there are other problems you can encounter. There was a thread either here or in centos-virt some time back about the other issues that can occur. Started by me. The solution for the other problems is to do your own bridging and disable the network-bridge script of Xen.
Kai
Brett Serkez wrote on Tue, 4 Aug 2009 08:43:28 -0400:
My understanding of the HWaddr is that the first portion is manufacturer assigned for uniqueness, I cannot image this NIC originally had this HWaddr, but I don't know what it originally was.
Indeed, AFAIK all hardware adapters start with 00. This must have been set in the BIOS or with a boot option or in the network config.
Kai
Brett Serkez wrote:
- Investigation concluded the issue was that the HWaddr address of the
physical NIC matched the fabricated HWaddr that XEN uses for most of its adapters: FE:FF:FF:FF:FF:FF.
I had the same problem after (IIRC) a kernel panic. After a few rounds of research and ineffective treatments I shut down the computer, left it unplugged for a few minutes, and turned it back on. The MAC went back to normal and it hasn't been an issue since.
Hope that helps.
-
Alan Sparks -
Andrew Norris -
Brett Serkez -
Christoph Maser -
Kai Schaetzl -
Lanny Marcus -
Mathew S. McCarrell -
Ross Walker -
Sergey Smirnov