Hi List,
I'm trying to configure two switches to provide redundancy (i.e. in case one switch goes down), and am wondering if there is a standard way to configure a CentOS box to use different gateways in a bonded interface, depending upon which physical nic is being used?
A bit more detail might help answer the "And why do you want to do that?" questions...
- Switch 1, Port 1: has an uplink to upstream provider, and is in VLAN-1 - Switch 2, Port 1: has a 2nd uplink to upstream provider, and is in VLAN-1 - Switch 1, Port 25+26 are trunked (link aggregation; depends on your terminology; these are procurve 2600's) and connected to Switch 2, Port 25+26. All vlans configured run on the trunked link, with tagging. - Uplinks to provider are running HSRP. Upstream provider creates a virtual IP (say, 2.2.2.1) that is the gateway to the outside world on one of the links, randomly. - Server A is connected to Switch 1, Port 2 on eth0; and Switch 2, Port 2 on eth1. Server is assigned an IP address, say, 1.1.1.2, inside VLAN-2 (1.1.1.0/24) - Server B, C, etc. exist in different VLANs. (Just saying this so as to avoid the suggestion of "don't use vlans" because that would make this mess go away, but create a security problem.) - Switch 1 does light routing. For VLAN-2, it is listening on, say, 1.1.1.254. - Switch 2 also does light routing. For VLAN-2, it is listening on, say, 1.1.1.253.
So, Server A's eth0 should use 1.1.1.254 as its gateway; and Server A's eth1 should use 1.1.1.253 as its gateway. The HSRP link will create 2.2.2.1 and an associated MAC address on one of its links; the two switches will throw the packets to the HSRP link on the correct switch. It doesn't matter which router (1.1.1.254 of 1.1.1.253) is used, since they will both route packets to the appropriate VLAN.
If one of the switches goes down, the surviving HSRP link will claim the virtual IP (2.2.2.1) and the switch will receive an arp to let it know to update its arp table. (I've also set mac and arp timeouts to 1 minute.)
So, the question remains... is there a way to tell linux that, given a bonded interface, the gateway for one physical nic is X while for a second physical nic it is Y? (Bonding can be active/standby.)
(If I had tons of $'s; I would just go buy some Cisco gear that's capable of creating a virtual chassis and be done with this. I don't have tons of $'s. If there are other ways of configuring this; I'd love to know; but from everything I can gather, passing the various VLAN traffic between the two switches (via tagged-VLANs on the interswitch connection) seems the only viable solution. Or should I really be looking at XRRP? Or Stacking? But that's a different list...)
How do people normally do this? (This seems too complicated to be right.)
best, Jeff
Jeff Potter wrote:
Hi List,
I'm trying to configure two switches to provide redundancy (i.e. in case one switch goes down), and am wondering if there is a standard way to configure a CentOS box to use different gateways in a bonded interface, depending upon which physical nic is being used?
You should google for "high availability Linux" and similar.
Possibly, heartbeat is part of the solution.
Do you have 2 gateways? Do you can't make it HA with VRRP?
----- Mensaje Original ----- De: "John Summerfield" debian@herakles.homelinux.org Para: "CentOS mailing list" centos@centos.org Enviados: lunes 26 de marzo de 2007 21H57 (GMT-0400) America/Santiago Asunto: Re: [CentOS] network redundancy via two nics, two routers?
Jeff Potter wrote:
Hi List,
I'm trying to configure two switches to provide redundancy (i.e. in case one switch goes down), and am wondering if there is a standard way to configure a CentOS box to use different gateways in a bonded interface, depending upon which physical nic is being used?
You should google for "high availability Linux" and similar.
Possibly, heartbeat is part of the solution.
Jeff Potter wrote:
Hi List,
I'm trying to configure two switches to provide redundancy (i.e. in case one switch goes down), and am wondering if there is a standard way to configure a CentOS box to use different gateways in a bonded interface, depending upon which physical nic is being used?
A bit more detail might help answer the "And why do you want to do that?" questions...
- Switch 1, Port 1: has an uplink to upstream provider, and is in VLAN-1
- Switch 2, Port 1: has a 2nd uplink to upstream provider, and is in VLAN-1
- Switch 1, Port 25+26 are trunked (link aggregation; depends on your
terminology; these are procurve 2600's) and connected to Switch 2, Port 25+26. All vlans configured run on the trunked link, with tagging.
- Uplinks to provider are running HSRP. Upstream provider creates a
virtual IP (say, 2.2.2.1) that is the gateway to the outside world on one of the links, randomly.
- Server A is connected to Switch 1, Port 2 on eth0; and Switch 2, Port
2 on eth1. Server is assigned an IP address, say, 1.1.1.2, inside VLAN-2 (1.1.1.0/24)
- Server B, C, etc. exist in different VLANs. (Just saying this so as to
avoid the suggestion of "don't use vlans" because that would make this mess go away, but create a security problem.)
- Switch 1 does light routing. For VLAN-2, it is listening on, say,
1.1.1.254.
- Switch 2 also does light routing. For VLAN-2, it is listening on, say,
1.1.1.253.
So, Server A's eth0 should use 1.1.1.254 as its gateway; and Server A's eth1 should use 1.1.1.253 as its gateway. The HSRP link will create 2.2.2.1 and an associated MAC address on one of its links; the two switches will throw the packets to the HSRP link on the correct switch. It doesn't matter which router (1.1.1.254 of 1.1.1.253) is used, since they will both route packets to the appropriate VLAN.
If one of the switches goes down, the surviving HSRP link will claim the virtual IP (2.2.2.1) and the switch will receive an arp to let it know to update its arp table. (I've also set mac and arp timeouts to 1 minute.)
So, the question remains... is there a way to tell linux that, given a bonded interface, the gateway for one physical nic is X while for a second physical nic it is Y? (Bonding can be active/standby.)
(If I had tons of $'s; I would just go buy some Cisco gear that's capable of creating a virtual chassis and be done with this. I don't have tons of $'s. If there are other ways of configuring this; I'd love to know; but from everything I can gather, passing the various VLAN traffic between the two switches (via tagged-VLANs on the interswitch connection) seems the only viable solution. Or should I really be looking at XRRP? Or Stacking? But that's a different list...)
How do people normally do this? (This seems too complicated to be right.)
Should it not be that there would be a third ip that would be set as the gateway and the two routers will claim the ip as needed? So it would be the same ip used as the gateway whether the packets go out through eth0 or eth1. As for bonded interface...should not that be on the same switch? You have a bonded interface on a VLAN that uses ports on two separate switches?
Hi John, Patricio, Feizhou -- thanks for your thoughts.
John, heartbeat looks like a viable solution; I hadn't considered it for "just" monitoring the NICs, but ipfail -- http://www.linux-ha.org/ ConfigureIpfail -- seems to suggest that it will at least allow for a basic "ping gateway 1; if it's down, switch to gateway 2" setup. (I had considered just writing this myself; but something tells me that there are probably subtle gotchas that I would miss. Not to mention reinventing the wheel.) Thanks!
I'm curious, though, if there might not be a lower-level way of doing this, so as to not rely on the heartbeat process running? (i.e. "K.I.S.S.")
Should it not be that there would be a third ip that would be set as the gateway and the two routers will claim the ip as needed? So it would be the same ip used as the gateway whether the packets go out through eth0 or eth1.
Feizhou, that would be ideal... but I don't know of any way in the HP ProCurve switches of doing this. Each switch would have to health- check the other, and the standby would have to pick up that IP if the primary went down. They don't seem to support this. That would be 100% perfect, though; because in that setup, CentOS doesn't have to even know that the bonded interface is split between two physical switches or that the gateway is relocatable.
(Patricio: I will look into VRRP/XRRP again; maybe it will solve this? Thank you for the suggestion!)
As for bonded interface...should not that be on the same switch? You have a bonded interface on a VLAN that uses ports on two separate switches?
Yes; on separate switches; which are connected and know to shuffle packets back and forth for the VLANs. If the bonded connections went to the same switch, and that switch went offline, then... well, it doesn't solve the problem of eliminating switch or power failure as a cause of downtime. (Switches will be plugged into different power grids.)
So the question remains: besides installing software that health- checks the connection, is there a way to configure linux bonding to use different gateways based on the physical NIC in a bonded pair? (Or, a way to tell it to try two different gateways; and if one is down, to try the other? I.e. some sort of route cost solution?)
best, Jeff
Should it not be that there would be a third ip that would be set as the gateway and the two routers will claim the ip as needed? So it would be the same ip used as the gateway whether the packets go out through eth0 or eth1.
Feizhou, that would be ideal... but I don't know of any way in the HP ProCurve switches of doing this. Each switch would have to health-check the other, and the standby would have to pick up that IP if the primary went down. They don't seem to support this. That would be 100% perfect, though; because in that setup, CentOS doesn't have to even know that the bonded interface is split between two physical switches or that the gateway is relocatable.
Oh sorry, I read your post wrong. So you have:
link1 link2 2.2.2.1 HSRP VLAN1 Procurve1 Procurve2 VLAN2 1.1.1.254 1.1.1.253 1.1.1.2 BONDED ServerA
I am not sure but I think you could use iproute2 tools.
Create two routing tables. Add for example the below to /etc/iproute2/rt_tables:
10 switchA 11 switchB
Create a script or put the commands to populate those routing tables into rc.local
ip route add 1.1.1.0/24 dev eth0 table switchA ip route add default via 1.1.1.254 dev eth0 table switchA ip route add 1.1.1.0/24 dev eth1 table switchB ip route add default via 1.1.1.253 dev eth1 table switchB
Below are commands that make the system lookup appropriate routing tables depending on interface. ip rule add from 1.1.1.2 dev eth0 lookup switchA ip rule add from 1.1.1.2 dev eth1 lookup switchB
The question is whether using dev ethX will be honoured...as I understand that you will get an interface called bond0...
So the question remains: besides installing software that health-checks the connection, is there a way to configure linux bonding to use different gateways based on the physical NIC in a bonded pair? (Or, a way to tell it to try two different gateways; and if one is down, to try the other? I.e. some sort of route cost solution?)
Or maybe you could have a regular check on the link status with mii-tool ... if eth0 is down, use eth1 gw as default route and vice versa.
-
Feizhou -
Jeff Potter -
John Summerfield -
Patricio A. Bruna