I have nic bonding (mode=802.3ad) setup on 2 servers, both running Centos 5.5 In the "Active Aggregator Info", on one reports 4 ports - which is correct - but the other only reports 3 ports. It's always eth0 that shows a different aggregator ID. Changing the cables around so it hits a different port on the switch makes no difference. The switch is correctly configured for the port channel. The only difference between the 2 servers is one is running Xen (kernel-xen-2.6.18-194.32.1.el5.centos.plus) so I have a few virtual interfaces and bridges.
Any ideas what might be happening?
Thanx,
Russell Smithies
[root@invmhp01 ~]# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.4.0 (October 7, 2008)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation Transmit Hash Policy: layer2 (0) MII Status: up MII Polling Interval (ms): 100 Up Delay (ms): 0 Down Delay (ms): 0
802.3ad info LACP rate: slow Active Aggregator Info: Aggregator ID: 19 Number of ports: 3 <<<<<<<< Actor Key: 17 Partner Key: 5 Partner Mac Address: 00:1b:90:3d:90:c0
Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5c Aggregator ID: 18
Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5d Aggregator ID: 19
Slave Interface: eth2 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5e Aggregator ID: 19
Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5f Aggregator ID: 19
[root@invmhp01 ~]# ifconfig bond0 Link encap:Ethernet HWaddr 1C:C1:DE:74:97:5C inet addr:147.158.130.183 Bcast:147.158.131.255 Mask:255.255.252.0 inet6 addr: fe80::1ec1:deff:fe74:975c/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:64087 errors:0 dropped:0 overruns:0 frame:0 TX packets:18179 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:8293500 (7.9 MiB) TX bytes:31362799 (29.9 MiB)
eth0 Link encap:Ethernet HWaddr 1C:C1:DE:74:97:5C UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:5770 errors:0 dropped:0 overruns:0 frame:0 TX packets:21 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:410102 (400.4 KiB) TX bytes:2524 (2.4 KiB)
eth1 Link encap:Ethernet HWaddr 1C:C1:DE:74:97:5C UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:22247 errors:0 dropped:0 overruns:0 frame:0 TX packets:17436 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1424685 (1.3 MiB) TX bytes:31161969 (29.7 MiB) Interrupt:239
eth2 Link encap:Ethernet HWaddr 1C:C1:DE:74:97:5C UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:6223 errors:0 dropped:0 overruns:0 frame:0 TX packets:189 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:442326 (431.9 KiB) TX bytes:19993 (19.5 KiB) Interrupt:235
eth3 Link encap:Ethernet HWaddr 1C:C1:DE:74:97:5C UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:29852 errors:0 dropped:0 overruns:0 frame:0 TX packets:537 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:6016657 (5.7 MiB) TX bytes:179281 (175.0 KiB) Interrupt:231
lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:26818 errors:0 dropped:0 overruns:0 frame:0 TX packets:26818 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:38558929 (36.7 MiB) TX bytes:38558929 (36.7 MiB)
peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:41498 errors:0 dropped:0 overruns:0 frame:0 TX packets:251 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:3000012 (2.8 MiB) TX bytes:59877 (58.4 KiB) Interrupt:243
vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:127 errors:0 dropped:0 overruns:0 frame:0 TX packets:39354 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:15520 (15.1 KiB) TX bytes:2817598 (2.6 MiB)
virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:42 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:8160 (7.9 KiB)
xenbr0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:38746 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:2217067 (2.1 MiB) TX bytes:0 (0.0 b)
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
On 16/02/2011, at 3:12 PM, Smithies, Russell wrote:
802.3ad info LACP rate: slow Active Aggregator Info: Aggregator ID: 19 Number of ports: 3 <<<<<<<< Actor Key: 17 Partner Key: 5 Partner Mac Address: 00:1b:90:3d:90:c0
Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5c Aggregator ID: 18 <<<<<<<<<
This is different.
Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5d Aggregator ID: 19
Slave Interface: eth2 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5e Aggregator ID: 19
Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5f Aggregator ID: 19
Yep, that's the problem - it keeps coming up with 3 ports instead of 4 and eth0 is always has a different Aggregator ID. No idea why it does that - the other server is setup the same and it's bonding works perfectly. The only thing I can think of is some of the Xen virtual interfaces and bridges as disrupting it.
--Russell
From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Cameron Kerr Sent: Thursday, 17 February 2011 6:18 a.m. To: CentOS mailing list Subject: Re: [CentOS] NIC bonding - missing eth0?
On 16/02/2011, at 3:12 PM, Smithies, Russell wrote:
802.3ad info LACP rate: slow Active Aggregator Info: Aggregator ID: 19 Number of ports: 3 <<<<<<<< Actor Key: 17 Partner Key: 5 Partner Mac Address: 00:1b:90:3d:90:c0
Slave Interface: eth0 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5c Aggregator ID: 18 <<<<<<<<<
This is different.
Slave Interface: eth1 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5d Aggregator ID: 19
Slave Interface: eth2 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5e Aggregator ID: 19
Slave Interface: eth3 MII Status: up Link Failure Count: 0 Permanent HW addr: 1c:c1:de:74:97:5f Aggregator ID: 19
======================================================================= Attention: The information contained in this message and/or attachments from AgResearch Limited is intended only for the persons or entities to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipients is prohibited by AgResearch Limited. If you have received this message in error, please notify the sender immediately. =======================================================================
On 17/02/2011, at 8:40 AM, Smithies, Russell wrote:
Yep, that’s the problem – it keeps coming up with 3 ports instead of 4 and eth0 is always has a different Aggregator ID. No idea why it does that – the other server is setup the same and it’s bonding works perfectly.
I would think then it might be switch related. What brand/model of switch do you have?
Looking at the Wikipedia page for Link aggregation[1], it seems that 802.3ad mode creates groups from ports that share the same speed and duplex settings. There is also an implication that support has to be present on the switch, so I certainly wouldn't rule that out.
[1] http://en.wikipedia.org/wiki/Link_aggregation
With regard to your Xen instances, ensure that there is only interacting with bondX interfaces, and not ethX. Check inside /etc/xen/vif-bridge and network-bridge... bridging could certainly be a factor so you ought to take down your Xen instances and see if it works.
Also, can you see any LACP frames going in and out of all of the interfaces?
-
Cameron Kerr -
Smithies, Russell