We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
Any ideas ?
07:38:15.480141 802.1Q vlan#911 P0 10.90.11.244.http > 10.90.11.1.47970: . ack 9 win 1098 (DF) 07:38:15.691074 802.1Q vlan#911 P0 10.10.0.27.34023 > 10.90.11.244.http: S 2715145345:2715145345(0) win 4380 <mss 1460,nop,wscale 0,nop,nop,timestamp 72820060 0,sackOK,eol> (DF) 07:38:15.691074 802.1Q vlan#902 P0 10.90.11.244.http > 10.10.0.27.34023: S 514059651:514059651(0) ack 2715145346 win 5808 <mss 1464,sackOK,timestamp 51754641 72820060,nop,wscale 2> (DF) 07:38:18.630101 802.1Q vlan#902 P0 10.90.11.244.http > 10.10.0.27.34023: S 514059651:514059651(0) ack 2715145346 win 5808 <mss 1464,sackOK,timestamp 51757641 72820060,nop,wscale 2> (DF) 07:38:18.630101 802.1Q vlan#911 P0 10.10.0.27.34023 > 10.90.11.244.http: S 2715145345:2715145345(0) win 4380 <mss 1460,nop,wscale 0,nop,nop,timestamp 72823060 0,sackOK,eol> (DF) 07:38:18.630101 802.1Q vlan#902 P0 10.90.11.244.http > 10.10.0.27.34023: S 514059651:514059651(0) ack 2715145346 win 5808 <mss 1464,sackOK,timestamp 51757642 72820060,nop,wscale 2> (DF) 07:38:20.520099 802.1Q vlan#911 P0 10.90.11.1.48003 > 10.90.11.244.http: S 1650014488:1650014488(0) win 5840 <mss 1460,nop,nop,sackOK,nop,wscale 0> (DF)
On Fri, 2005-10-28 at 09:48, Robin Mordasiewicz wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
It seems reasonable to require a route to the destination on the VLAN used. Why should it ever do otherwise? What are you trying to accomplish by using a VLAN interface with no route back?
The solution should be just the same as it would be with real interfaces which is to have the BigIP source-NAT the inbound requests into an address on that VLAN, but you won't like that because your web logs won't show the real client addresses. Basically the Linux box needs some reason to send the reply packet back to the BigIP either on your chosen VLAN or a physical interface. That can either be because it appears to have come from there (SNAT), or because it is the default gateway.
On Fri, 28 Oct 2005, Les Mikesell wrote:
On Fri, 2005-10-28 at 09:48, Robin Mordasiewicz wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
It seems reasonable to require a route to the destination on the VLAN used. Why should it ever do otherwise? What are you trying to accomplish by using a VLAN interface with no route back?
Is there any way to say that if traffic is recieved on VLAN#911 to be sure that the return traffic is tagged with the same vlan id. Currently traffic is tagged based on the routing table, and even if traffic comes in on VLAN#911, when it returns the traffc it uses the VLAN tag from the network that the default gateway is on(VLAN#902).
Quoting Robin Mordasiewicz robin@bullseye.tv:
Is there any way to say that if traffic is recieved on VLAN#911 to be sure that the return traffic is tagged with the same vlan id. Currently traffic is tagged based on the routing table, and even if traffic comes in on VLAN#911, when it returns the traffc it uses the VLAN tag from the network that the default gateway is on(VLAN#902).
If you can use something to identify those outgoing packets inside Netfilter, you can set firewall mark on them, and than use that firewall mark to route packets to correct VLAN. For example, if all port 80 traffic is from VLAN#911, and there is no way for traffic from VLAN#902 (or anywhere else) to get to your box, this might work.
Failing that, you might consider CONNMARK target. Than you could set connection mark on the incomming packet (hm, is there a way to set mark based n VLAN tag?), and then based on that set firewall mark on the outgoing packets (--save-mark and --restore-mark options of CONNMARK target). And than, again, route outgoing packets based on the firewall mark. I'm not sure if CONNMARK target is included with CentOS kernel.
---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On 10/28/05, Robin Mordasiewicz robin@bullseye.tv wrote:
On Fri, 28 Oct 2005, Les Mikesell wrote:
On Fri, 2005-10-28 at 09:48, Robin Mordasiewicz wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
It seems reasonable to require a route to the destination on the VLAN used. Why should it ever do otherwise? What are you trying to accomplish by using a VLAN interface with no route back?
Is there any way to say that if traffic is recieved on VLAN#911 to be sure that the return traffic is tagged with the same vlan id. Currently traffic is tagged based on the routing table, and even if traffic comes in on VLAN#911, when it returns the traffc it uses the VLAN tag from the network that the default gateway is on(VLAN#902).
You might be able to use iptables "mark" capability or you would need to see if CentOS supports policy based routing based on inbound interface, source MAC, or source IP, all of the of the original packet.
-- Leonard Isham, CISSP Ostendo non ostento.
On Fri, 2005-10-28 at 11:14, Robin Mordasiewicz wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
It seems reasonable to require a route to the destination on the VLAN used. Why should it ever do otherwise? What are you trying to accomplish by using a VLAN interface with no route back?
Is there any way to say that if traffic is recieved on VLAN#911 to be sure that the return traffic is tagged with the same vlan id. Currently traffic is tagged based on the routing table, and even if traffic comes in on VLAN#911, when it returns the traffc it uses the VLAN tag from the network that the default gateway is on(VLAN#902).
The BigIP will do this sort of magic itself to save the time looking up the return route, but it really is black magic in terms of standard networking where asymmetrical routes are permitted and expected. The reply packet doesn't have much to connect it to the one that came in and it's path is determined by the route to the destination address. That said, there may be some black magic you can do with iptables and the ip_conntrack info or some sort of policy based routing.
On Fri, 28 Oct 2005, Les Mikesell wrote:
On Fri, 2005-10-28 at 11:14, Robin Mordasiewicz wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
It seems reasonable to require a route to the destination on the VLAN used. Why should it ever do otherwise? What are you trying to accomplish by using a VLAN interface with no route back?
Is there any way to say that if traffic is recieved on VLAN#911 to be sure that the return traffic is tagged with the same vlan id. Currently traffic is tagged based on the routing table, and even if traffic comes in on VLAN#911, when it returns the traffc it uses the VLAN tag from the network that the default gateway is on(VLAN#902).
The BigIP will do this sort of magic itself to save the time looking up the return route, but it really is black magic in terms of standard networking where asymmetrical routes are permitted and expected. The reply packet doesn't have much to connect it to the one that came in and it's path is determined by the route to the destination address. That said, there may be some black magic you can do with iptables and the ip_conntrack info or some sort of policy based routing.
I will research policy based routing.
On Fri, 28 Oct 2005, Robin Mordasiewicz wrote:
On Fri, 28 Oct 2005, Les Mikesell wrote:
On Fri, 2005-10-28 at 11:14, Robin Mordasiewicz wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
It seems reasonable to require a route to the destination on the VLAN used. Why should it ever do otherwise? What are you trying to accomplish by using a VLAN interface with no route back?
Is there any way to say that if traffic is recieved on VLAN#911 to be sure that the return traffic is tagged with the same vlan id. Currently traffic is tagged based on the routing table, and even if traffic comes in on VLAN#911, when it returns the traffc it uses the VLAN tag from the network that the default gateway is on(VLAN#902).
The BigIP will do this sort of magic itself to save the time looking up the return route, but it really is black magic in terms of standard networking where asymmetrical routes are permitted and expected. The reply packet doesn't have much to connect it to the one that came in and it's path is determined by the route to the destination address. That said, there may be some black magic you can do with iptables and the ip_conntrack info or some sort of policy based routing.
I will research policy based routing.
This is now working properly. After googling "policy based routing linux" I came across a very helpful article, which explained how to use the ip command to create multiple routing tables. I have ended up creating a routing table for each tagged VLAN.
http://www.linuxjournal.com/article/7291
My setup is slightly different than the article, as I am bonding nic's together and then bringing up 802.1q tagged interfaces on the bond.
I have a multi homed, as in multiple tagged vlan interfaces. For example I have an interface bond0.101(10.10.1.244), bond0.102(10.10.2.244), bond0.103(10.10.3.244). For each interface I have issued commands...
ip ro add 0.0.0.0/0 via 10.10.1.1 table 101 ip ru add from 10.10.1.244 table 101 ip ro add 0.0.0.0/0 via 10.10.2.1 table 102 ip ru add from 10.10.2.244 table 102 ip ro add 0.0.0.0/0 via 10.10.3.1 table 103 ip ru add from 10.10.3.244 table 103
Now each interface has its own routing table. I may have left out some information in my explanation, as I am still teaching myself how to use the ip command, and to configure the /etc/iproute2/* config files.
Any further hints are appreciated.
On Fri, 28 Oct 2005, Robin Mordasiewicz wrote:
On Fri, 28 Oct 2005, Robin Mordasiewicz wrote:
On Fri, 28 Oct 2005, Les Mikesell wrote:
On Fri, 2005-10-28 at 11:14, Robin Mordasiewicz wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
It seems reasonable to require a route to the destination on the VLAN used. Why should it ever do otherwise? What are you trying to accomplish by using a VLAN interface with no route back?
Is there any way to say that if traffic is recieved on VLAN#911 to be sure that the return traffic is tagged with the same vlan id. Currently traffic is tagged based on the routing table, and even if traffic comes in on VLAN#911, when it returns the traffc it uses the VLAN tag from the network that the default gateway is on(VLAN#902).
The BigIP will do this sort of magic itself to save the time looking up the return route, but it really is black magic in terms of standard networking where asymmetrical routes are permitted and expected. The reply packet doesn't have much to connect it to the one that came in and it's path is determined by the route to the destination address. That said, there may be some black magic you can do with iptables and the ip_conntrack info or some sort of policy based routing.
I will research policy based routing.
This is now working properly. After googling "policy based routing linux" I came across a very helpful article, which explained how to use the ip command to create multiple routing tables. I have ended up creating a routing table for each tagged VLAN.
http://www.linuxjournal.com/article/7291
My setup is slightly different than the article, as I am bonding nic's together and then bringing up 802.1q tagged interfaces on the bond.
I have a multi homed, as in multiple tagged vlan interfaces. For example I have an interface bond0.101(10.10.1.244), bond0.102(10.10.2.244), bond0.103(10.10.3.244). For each interface I have issued commands...
ip ro add 0.0.0.0/0 via 10.10.1.1 table 101 ip ru add from 10.10.1.244 table 101 ip ro add 0.0.0.0/0 via 10.10.2.1 table 102 ip ru add from 10.10.2.244 table 102 ip ro add 0.0.0.0/0 via 10.10.3.1 table 103 ip ru add from 10.10.3.244 table 103
Now each interface has its own routing table. I may have left out some information in my explanation, as I am still teaching myself how to use the ip command, and to configure the /etc/iproute2/* config files.
Any further hints are appreciated.
This is also a must read for any Linux sysadmin, which needs to learn more about more advanced networking http://linux-ip.net/
On 10/28/05, Robin Mordasiewicz robin@bullseye.tv wrote:
We are using Centos behind an F5 Bigip load balancer. The linux box is using bonding and tagged VLAN's
Everything works fine except that when traffic is forwarded from the BigIP to the linux box on the VLAN where the web server is running the linux box returns the traffic on the wrong VLAN, It returns traffic on the lowest ordered VLAN.
ie. here is a tcpdump on my load balancer showing traffic being sent on VLAN 911 to the linux box, but the linux box returns traffic on VLAN 902. The linux box is returning traffic on the same VLAN as its configured default gateway. If I change the default gateway to be on the VLAN 911 then everytyhing works.
Any ideas ?
07:38:15.480141 802.1Q vlan#911 P0 10.90.11.244.http > 10.90.11.1.47970: . ack 9 win 1098 (DF) 07:38:15.691074 802.1Q vlan#911 P0 10.10.0.27.34023 > 10.90.11.244.http: S 2715145345:2715145345(0) win 4380 <mss 1460,nop,wscale 0,nop,nop,timestamp 72820060 0,sackOK,eol> (DF) 07:38:15.691074 802.1Q vlan#902 P0 10.90.11.244.http > 10.10.0.27.34023: S 514059651:514059651(0) ack 2715145346 win 5808 <mss 1464,sackOK,timestamp 51754641 72820060,nop,wscale 2> (DF)
[snip]
If you are talking about the traffic with 10.10.0.27.34023 then it appears to me that you are dealing with a standard routing issue.
I have not done this on Linux, but on other systems, including routers.
Each VLAN is treated as a separate interface and packets are returned based on routing tables not the interface they where received on. That is unless you are using policy based routing.
-- Leonard Isham, CISSP Ostendo non ostento.
-
Aleksandar Milivojevic -
Leonard Isham -
Les Mikesell -
Robin Mordasiewicz