I have a number of machines that have 4 NICs, two of which are actually in use, running Centos 5. When they are rebooted, they seem to change the eth interface names, assigning them in different orders. I'm a little fuzzy on the details because they are at a remote location and I can't access them easily - especially after the network breaks. Shouldn't: alias eth0 bnx2 alias eth1 bnx2 alias eth2 e1000 alias eth3 e1000 in /etc/modprobe.conf always make the intel cards eth2 and 3?
Les Mikesell wrote:
I have a number of machines that have 4 NICs, two of which are actually in use, running Centos 5. When they are rebooted, they seem to change the eth interface names, assigning them in different orders. I'm a little fuzzy on the details because they are at a remote location and I can't access them easily - especially after the network breaks. Shouldn't: alias eth0 bnx2 alias eth1 bnx2 alias eth2 e1000 alias eth3 e1000 in /etc/modprobe.conf always make the intel cards eth2 and 3?
Noop, this is done with ifcfg-ethX where X is the if number.
Create a /etc/sysconfig/network/ifcfg-eth0 that look like this example:
DEVICE=eth0 HWADDR=00:01:23:45:67:89 ONBOOT=yes TYPE=Ethernet NETMASK=255.255.255.0 IPADDR=192.168.1.154 GATEWAY=192.168.1.1
and then create ifcfg-eth1, ifcfg-eth2, ifcfg-eth3
then do a "service network restart" to activate the settings.
/Mats
MatsK wrote:
Les Mikesell wrote:
I have a number of machines that have 4 NICs, two of which are actually in use, running Centos 5. When they are rebooted, they seem to change the eth interface names, assigning them in different orders. I'm a little fuzzy on the details because they are at a remote location and I can't access them easily - especially after the network breaks. Shouldn't: alias eth0 bnx2 alias eth1 bnx2 alias eth2 e1000 alias eth3 e1000 in /etc/modprobe.conf always make the intel cards eth2 and 3?
Noop, this is done with ifcfg-ethX where X is the if number.
Create a /etc/sysconfig/network/ifcfg-eth0 that look like this example:
DEVICE=eth0 HWADDR=00:01:23:45:67:89 ONBOOT=yes TYPE=Ethernet NETMASK=255.255.255.0 IPADDR=192.168.1.154 GATEWAY=192.168.1.1
and then create ifcfg-eth1, ifcfg-eth2, ifcfg-eth3
then do a "service network restart" to activate the settings.
I do have the ifcfg-ethX files for the 2 interfaces that are currently active, but since the machines were built by image copies of a master disk, they do not have HWADDR address entries. A person on-site with access to the console adjusted them if they didn't come up right the first time, but they seem to shift around on each reboot. Will adding the HWADDR entry nail them down even if it doesn't match the nic type specified in modprobe.conf? Can someone point me to the code where this happens? Until recently the machines were running centos 3.x and this seems to be a difference in behavior.
MatsK wrote:
Les Mikesell wrote:
I have a number of machines that have 4 NICs, two of which are actually in use, running Centos 5. When they are rebooted, they seem to change the eth interface names, assigning them in different orders. I'm a little fuzzy on the details because they are at a remote location and I can't access them easily - especially after the network breaks. Shouldn't: alias eth0 bnx2 alias eth1 bnx2 alias eth2 e1000 alias eth3 e1000 in /etc/modprobe.conf always make the intel cards eth2 and 3?
Noop, this is done with ifcfg-ethX where X is the if number.
Create a /etc/sysconfig/network/ifcfg-eth0 that look like this example:
DEVICE=eth0 HWADDR=00:01:23:45:67:89 ONBOOT=yes TYPE=Ethernet NETMASK=255.255.255.0 IPADDR=192.168.1.154 GATEWAY=192.168.1.1
and then create ifcfg-eth1, ifcfg-eth2, ifcfg-eth3
then do a "service network restart" to activate the settings.
I do have the ifcfg-ethX files for the 2 interfaces that are currently active, but since the machines were built by image copies of a master disk, they do not have HWADDR address entries. A person on-site with access to the console adjusted them if they didn't come up right the first time, but they seem to shift around on each reboot. Will adding the HWADDR entry nail them down even if it doesn't match the nic type specified in modprobe.conf? Can someone point me to the code where this happens? Until recently the machines were running centos 3.x and this seems to be a difference in behavior.
In my experience, adding the HWADDR line to your ifcfg-ethX files will tie the network interface to the right card, regardless of modprobe.conf entries. I usually remove HWADDR lines when anaconda provides them at install time because if I replace a nic (which obviously has a different MAC address) without updating the HWADDR line, the interface fails to start. In cases where modprobe.conf is unable to order the interfaces as I want it to, I add HWADDR lines. Works every time.
Hope this helps.
Barry
Les Mikesell wrote:
MatsK wrote:
Les Mikesell wrote:
I have a number of machines that have 4 NICs, two of which are actually in use, running Centos 5. When they are rebooted, they seem to change the eth interface names, assigning them in different orders. I'm a little fuzzy on the details because they are at a remote location and I can't access them easily - especially after the network breaks. Shouldn't: alias eth0 bnx2 alias eth1 bnx2 alias eth2 e1000 alias eth3 e1000 in /etc/modprobe.conf always make the intel cards eth2 and 3?
Noop, this is done with ifcfg-ethX where X is the if number.
Create a /etc/sysconfig/network/ifcfg-eth0 that look like this example:
DEVICE=eth0 HWADDR=00:01:23:45:67:89 ONBOOT=yes TYPE=Ethernet NETMASK=255.255.255.0 IPADDR=192.168.1.154 GATEWAY=192.168.1.1
and then create ifcfg-eth1, ifcfg-eth2, ifcfg-eth3
then do a "service network restart" to activate the settings.
I do have the ifcfg-ethX files for the 2 interfaces that are currently active, but since the machines were built by image copies of a master disk, they do not have HWADDR address entries. A person on-site with access to the console adjusted them if they didn't come up right the first time, but they seem to shift around on each reboot. Will adding the HWADDR entry nail them down even if it doesn't match the nic type specified in modprobe.conf? Can someone point me to the code where this happens? Until recently the machines were running centos 3.x and this seems to be a difference in behavior.
the names in modprobe.conf don't mean much: you could have eth0 as an alias to your sound card module and snd-card-0 an alias to your ethernet module, except for timing problems on startup...
I assume the ifup scripts modprobe's the corresponding eth device, so if your alias is correct you can be sure the module will be loaded in time for activating the device. However, if your modprobe.conf has "alias eth1 bnx2" but eth1 is an intel NIC for some reason, ifup eth1 will still work fine and bring up the intel eth1 as long as the intel module (e1000?) has been loaded at that point.
make sense?
cheers,
Les Mikesell wrote:
I do have the ifcfg-ethX files for the 2 interfaces that are currently active, but since the machines were built by image copies of a master disk, they do not have HWADDR address entries. A person on-site with access to the console adjusted them if they didn't come up right the first time, but they seem to shift around on each reboot. Will adding the HWADDR entry nail them down even if it doesn't match the nic type specified in modprobe.conf? Can someone point me to the code where this happens? Until recently the machines were running centos 3.x and this seems to be a difference in behavior.
As already pointed out, yes adding HWADDR will "nail them down" and the entries in modprobe.conf don't mean much. If you (or a script) execute "modprobe eth0" it will load the appropriate module. Unfortunately, this is not how CentOS 5 loads drivers.
With CentOS 5, udev is used to load the drivers by looking at the "modalias" file found for each device under the /sys directory (search for them, there are many). For PCI devices, the modalias includes the 4 16-bit PCI ID values, the PCI device type, and some other information.
Unfortunately, udev tries to be clever and loads drivers in parallel. As a result, if there are NICs that use different drivers, the order that the NICs are assigned ethX interfaces is left to the whim of the Linux scheduler (i.e. is non-deterministic). Devices using the same driver will always be assigned interface names in the same relative ordering. If they all use the same driver, they will always be assigned the same names, without having to fuss with the HWADDR option (this is due to how drivers enumerate PCI devices).
In reality, HWADDR doesn't force the kernel to assign the desired interface to each device. It simply "cleans up" after udev by renaming the interfaces from what the kernel assigned to each NIC to the interfaces you expect. Search for "rename_device" in ifup-eth and network-functions, both found in the /etc/sysconfig/network-scripts directory.
Cheers, Michael
Les and Michael,
There are a few ways to workaround the NIC detection issue. Each has its own advantages and limits.
The first method is: suppose you or your team have full control of running kernel on your hundreds/thousands of boxes, your can then build some NIC drivers statically in the kernel -- these statically built NIC drivers will be detected as eth0 without glitches -- then leave other different NIC types on the same box still in dynamic kernel modules status. It works greatly if you know all the types of primary network NIC. Typically e100, tg3, etc. and you have already standardized the 2nd NIC on the boxes to one or two brands like e1000.
The second method is: suppose you or your team can not control rebuilding of kernel, or at least you have no full control, but you really know the types of primary/secondary NICs combinations on all the Linux boxes in your kingdom. Then you can try the following hack:
You can try to add/change lines in /lib/modules/`uname -r`/modules.dep file according to your NICs combinations -- always load the drivers according to your predefined order. For example:
.../e1000.ko: .../tg3.ko .../3c59x.ko .../e100.ko .../forcedeth.ko .../forcedeth.ko: .../tg3.ko
The above means to load the module at left, system will first load modules at right! So tg3|3c59x|e100|forcedeth always load before e1000, and tg3 load before forcedeth. The same idea can be applied to all NIC combination types your have and can be set only once and applied to all your linux boxes if you set it up correctly. The side-effect is: you have waste few hundreds Kilobytes memory, but who cares?
There are also other tricks I tried before, some works and some not. But I think the above should probably work for most general cases.
Have a good weekend.
--Guolin
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Michael D. Kralka Sent: Thursday, January 10, 2008 6:52 AM To: CentOS mailing list Subject: Re: [CentOS] Nic order detection
Les Mikesell wrote:
I do have the ifcfg-ethX files for the 2 interfaces that are currently active, but since the machines were built by image copies of a master disk, they do not have HWADDR address entries. A person on-site with access to the console adjusted them if they didn't come up right the first time, but they seem to shift around on each reboot. Will adding the HWADDR entry nail them down even if it doesn't match the nic type specified in modprobe.conf? Can someone point me to the code where
this
happens? Until recently the machines were running centos 3.x and this seems to be a difference in behavior.
As already pointed out, yes adding HWADDR will "nail them down" and the entries in modprobe.conf don't mean much. If you (or a script) execute "modprobe eth0" it will load the appropriate module. Unfortunately, this is not how CentOS 5 loads drivers.
With CentOS 5, udev is used to load the drivers by looking at the "modalias" file found for each device under the /sys directory (search for them, there are many). For PCI devices, the modalias includes the 4 16-bit PCI ID values, the PCI device type, and some other information.
Unfortunately, udev tries to be clever and loads drivers in parallel. As a result, if there are NICs that use different drivers, the order that the NICs are assigned ethX interfaces is left to the whim of the Linux scheduler (i.e. is non-deterministic). Devices using the same driver will always be assigned interface names in the same relative ordering. If they all use the same driver, they will always be assigned the same names, without having to fuss with the HWADDR option (this is due to how drivers enumerate PCI devices).
In reality, HWADDR doesn't force the kernel to assign the desired interface to each device. It simply "cleans up" after udev by renaming the interfaces from what the kernel assigned to each NIC to the interfaces you expect. Search for "rename_device" in ifup-eth and network-functions, both found in the /etc/sysconfig/network-scripts directory.
Cheers, Michael _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Guolin Cheng wrote:
Les and Michael,
I am going to bite my tongue and not ask to you refrain from top posting.
As your subject suggests, you are proposing a quick and dirty hack to deal with interface assignment to physical NICs. Why bother with a quick and dirty hack when a sensible solution exists within the distribution? I see this a bad advice and hope no one follows it.
There are a few ways to workaround the NIC detection issue. Each has its own advantages and limits.
The first method is: suppose you or your team have full control of running kernel on your hundreds/thousands of boxes, your can then build some NIC drivers statically in the kernel -- these statically built NIC drivers will be detected as eth0 without glitches -- then leave other different NIC types on the same box still in dynamic kernel modules status. It works greatly if you know all the types of primary network NIC. Typically e100, tg3, etc. and you have already standardized the 2nd NIC on the boxes to one or two brands like e1000.
Although this may "work", I have just signed up for a lifetime of chasing kernel versions. Every time RHEL/CentOS release a new kernel to fix a bug or security vulnerability, I must recompile the kernel. How does this make sense if I have hundreds/thousands of boxes to to keep up to date? I'd rather "yum update" on all the boxes (which is easy to do)
The second method is: suppose you or your team can not control rebuilding of kernel, or at least you have no full control, but you really know the types of primary/secondary NICs combinations on all the Linux boxes in your kingdom. Then you can try the following hack:
You can try to add/change lines in /lib/modules/`uname -r`/modules.dep file according to your NICs combinations -- always load the drivers according to your predefined order. For example:
.../e1000.ko: .../tg3.ko .../3c59x.ko .../e100.ko .../forcedeth.ko .../forcedeth.ko: .../tg3.ko
Although this may "work", it is another accident waiting to happen. This is a generated file and it is almost never a good idea to modify an generated file; one will get burned. I install a shiny new module that is not delivered as part of the kernel (drbd perhaps), and the post-install script runs "depmod -a" (a sensible thing to do); now I have just blown away the manual changes. Or ever time I install a new kernel (whether I am foolishly[1] building my own or using the distribution kernels), I have to remember to make this change. The worst part about this is that the effects will not be visible until the next time the server is rebooted (say 6 months when there is a power failure); the network interface assignment will be wrong. Good luck hunting down that problem in a pinch!
[1] Don't get me wrong, there is a time and a place for building custom kernels; this is just not one of them.
The above means to load the module at left, system will first load modules at right! So tg3|3c59x|e100|forcedeth always load before e1000, and tg3 load before forcedeth. The same idea can be applied to all NIC combination types your have and can be set only once and applied to all your linux boxes if you set it up correctly. The side-effect is: you have waste few hundreds Kilobytes memory, but who cares?
The problem is not the wasted memory, it's the fragility of its design.
There are also other tricks I tried before, some works and some not. But I think the above should probably work for most general cases.
Why resort to "tricks" when there is a perfectly good solution supported by the distribution? I've learned that it never pays to be clever. When resorting to neat little tricks to get things to work, they get forgotten, or worse when someone else must look into a problem, they spend most of the time trying to understand the clever way things are set up. When stability is a main concern, boring is always better.
Cheers, Michael
Michael D. Kralka wrote:
Why resort to "tricks" when there is a perfectly good solution supported by the distribution? I've learned that it never pays to be clever. When resorting to neat little tricks to get things to work, they get forgotten, or worse when someone else must look into a problem, they spend most of the time trying to understand the clever way things are set up. When stability is a main concern, boring is always better.
The problem is that the disk images are made in one location and swapped into place in others, by someone who knows hardware, not linux, so for a new machine we won't know the hardware address ahead of time. When I first realized that the NICs were detected in a different order I added a script that tried to bring them all up, look for link, assign an ip address and ping the associated router to figure out which 2 were in use and which address they should have. However I did not realize (and I still don't see this documented anywhere...) that the device names would be non-deterministic or that they could be renamed after the kernel assigns a name. I can probably tweak the script to pick up the mac address and include it in the ifcfg-ethX files to nail things down. But, I see something about adding udev rules for persistent names so this is probably going to change again.
Michael,
There are no points to argue about which are the best 'official' ways which just like a war between vi or Emacs before. I may be stupid but any methods fix users' problem are the best ones. I've tried the official 'rename' or udev ways before, but finally I gave up and end up the two ways I've mentioned. Espectially the seconds, it works perfectly when I rerolled my Centos 5.0 and 5.1 initrd.img files for custom Kickstart installation in a really large scale.
Good luck and have a new year.
--Guolin
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Michael D. Kralka Sent: Saturday, January 12, 2008 5:41 AM To: CentOS mailing list Subject: Re: a quick and dirty hack to 'fix' the problem in a large scale-- RE: [CentOS] Nic order detection
Guolin Cheng wrote:
Les and Michael,
I am going to bite my tongue and not ask to you refrain from top posting.
As your subject suggests, you are proposing a quick and dirty hack to deal with interface assignment to physical NICs. Why bother with a quick and dirty hack when a sensible solution exists within the distribution? I see this a bad advice and hope no one follows it.
There are a few ways to workaround the NIC detection issue. Each has
its
own advantages and limits.
The first method is: suppose you or your team have full control of running kernel on your hundreds/thousands of boxes, your can then
build
some NIC drivers statically in the kernel -- these statically built
NIC
drivers will be detected as eth0 without glitches -- then leave other different NIC types on the same box still in dynamic kernel modules status. It works greatly if you know all the types of primary network NIC. Typically e100, tg3, etc. and you have already standardized the
2nd
NIC on the boxes to one or two brands like e1000.
Although this may "work", I have just signed up for a lifetime of chasing kernel versions. Every time RHEL/CentOS release a new kernel to fix a bug or security vulnerability, I must recompile the kernel. How does this make sense if I have hundreds/thousands of boxes to to keep up to date? I'd rather "yum update" on all the boxes (which is easy to do)
The second method is: suppose you or your team can not control rebuilding of kernel, or at least you have no full control, but you really know the types of primary/secondary NICs combinations on all
the
Linux boxes in your kingdom. Then you can try the following hack:
You can try to add/change lines in /lib/modules/`uname
-r`/modules.dep
file according to your NICs combinations -- always load the drivers according to your predefined order. For example:
.../e1000.ko: .../tg3.ko .../3c59x.ko .../e100.ko .../forcedeth.ko .../forcedeth.ko: .../tg3.ko
Although this may "work", it is another accident waiting to happen. This is a generated file and it is almost never a good idea to modify an generated file; one will get burned. I install a shiny new module that is not delivered as part of the kernel (drbd perhaps), and the post-install script runs "depmod -a" (a sensible thing to do); now I have just blown away the manual changes. Or ever time I install a new kernel (whether I am foolishly[1] building my own or using the distribution kernels), I have to remember to make this change. The worst part about this is that the effects will not be visible until the next time the server is rebooted (say 6 months when there is a power failure); the network interface assignment will be wrong. Good luck hunting down that problem in a pinch!
[1] Don't get me wrong, there is a time and a place for building custom kernels; this is just not one of them.
The above means to load the module at left, system will first load modules at right! So tg3|3c59x|e100|forcedeth always load before
e1000,
and tg3 load before forcedeth. The same idea can be applied to all NIC combination types your have and can be set only once and applied to
all
your linux boxes if you set it up correctly. The side-effect is: you have waste few hundreds Kilobytes memory, but who cares?
The problem is not the wasted memory, it's the fragility of its design.
There are also other tricks I tried before, some works and some not.
But
I think the above should probably work for most general cases.
Why resort to "tricks" when there is a perfectly good solution supported by the distribution? I've learned that it never pays to be clever. When resorting to neat little tricks to get things to work, they get forgotten, or worse when someone else must look into a problem, they spend most of the time trying to understand the clever way things are set up. When stability is a main concern, boring is always better.
Cheers, Michael
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Les Mikesell wrote:
Can someone point me to the code where this happens? Until recently the machines were running centos 3.x and this seems to be a difference in behavior.
CentOS 3 and 4 uses dev structure but with 5 was udev introduced. Thats the reason for the changes, but if you uses the /etc/sysconfig/network-scripts will everything be fine. Read more about it in the RHEL manual.
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of MatsK Sent: Thursday, January 10, 2008 12:42 PM To: CentOS mailing list Subject: Re: [CentOS] Nic order detection
Les Mikesell wrote:
Can someone point me to the code where this happens? Until recently the machines were running centos 3.x and this
seems to be a difference in behavior.
CentOS 3 and 4 uses dev structure but with 5 was udev introduced. Thats the reason for the changes, but if you uses the /etc/sysconfig/network-scripts will everything be fine. Read more about it in the RHEL manual. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-
Barry Brimer -
Guolin Cheng -
Les Mikesell -
MatsK -
Michael D. Kralka -
Nicolas Thierry-Mieg