[CentOS] Configuring source-specific routing

Wed May 1 21:15:57 UTC 2013
Michael H. Warfield <mhw at WittsEnd.com>

On Wed, 2013-05-01 at 16:05 -0400, Michael Mol wrote:
> I'm attempting to configure source-specific routing so that my servers
> can exist on multiple subnets from multiple upstream providers.

Kinda curious why you are attempting this without getting involved in
dynamic routing (BGP)...  It's usually someone trying to do multihoming
or multi-link load balancing on the cheap without involving their ISPs
(which tends to be expensive as soon as you're talking with them about
redundant / backup loops, provider independent addresses, and BGP
peering).  Generally equates to "champagne taste on a beer budget" but
there are exceptions and reasons, as I know from personal experience.
It often doesn't end well and is unreliable as network conditions
change.  But that depends on your requirements and application.  I'm not
one to judge - just pointing out the pitfalls.

I have done this a number of times in the past (mostly for VPN's and
redundant load-balancing links).  You're probably going to have get real
down and dirty into policy routing rules and tables with iproute2.  I
don't honestly believe you will be able to pull it off with the basic
stuff provided in the ifcfg-*, route-*, or static-route files (proviso

I had to do it using completely custom files utilizing "ip rule" and "ip
route {add|delete} table [n]" subcommands to "ip" to build custom
matching rules and mapping them to different routing tables containing
different routes and priorities.  In some cases, with OpenVPN VPNs, I
also had to incorporate iptables filtering commands to mark and match
packets and interact with the ip rule tables but I doubt you're going
that deep.

man ip-rule

       In some circumstances we want to route packets differently depending
       not only on destination addresses, but also on other packet fields:
       source address, IP protocol, transport protocol ports or even packet
       payload.  This task is called 'policy routing'.

       To solve this task, the conventional destination based routing table,
       ordered according to the longest match rule, is replaced with a 'rout‐
       ing policy database' (or RPDB), which selects routes by executing some
       set of rules.

This is way beyond what the network-scripts were designed for and WAY
WAY beyond NetworkMangler's capabilities.  It can be done but it's
difficult to get right and very dependent on your exact peculiar
configuration.  It can be real twitchy to make work right and real
difficult to debug when it's not working right.  It's also prone to
asymmetrical / triangular routes that can break things if you don't get
it right.  It also will not behave robustly if one link or another goes
down (no automatic failover).

It's beyond my ability to tell you generically how to make it work (I
can't even find those old config files I use to use for multiple ISDN
links years ago).  The devil is in the (your) details and here there be

It will end up looking something like this (and these addresses are in
my real address blocks...):

ip addr add dev eth0
ip addr add dev eth0

ip route add table 16 default via
ip route add table 17 default via

ip rule add from table 16
ip rule add from table 17

Those are just from memory and may not be perfectly accurate.  I haven't
needed to do any of this in years.  It's just possible that the above
commands could be incorporated into the route-* files.  Last time I
tried this was before iproute2 was fully integrated and I could only use
the static-routes file, which used the "route" command and would NOT
work.  Since the route-* files use the ip (iproute2) command, you should
be able to integrate the appropriate rule and route add commands into
those files, but I've not done it that way so I can't really vouch for

You can then see your match rules with this command:

[root at canyon ~]# ip rule ls
0:	from all lookup local 
32764:	from lookup 17 
32765:	from lookup 16 
32766:	from all lookup main 
32767:	from all lookup default 

And your route tables (including my default) like this:

[root at complex ~]# ip route ls table 16
default via dev eth0 
[root at complex ~]# ip route ls table 17
default via dev eth0 
[root at complex ~]# ip route ls
default via dev eth2 dev eth2  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev eth0  proto kernel  scope link  src dev br0  proto kernel  scope link  src dev eth2  scope link  metric 1003 dev br0  scope link  metric 1007 

There MAY be other ways of doing it but this was what use to work for
me, when I needed it many ages ago.


> A rough diagram of the network layout:

> ISP1 router (blackbox, routes subnet A, address on subnet A)
>   \
>    -----------eth0(firewall)eth1---((servers))
>   /
> ISP2 router (blackbox, routes subnet B, address on subnet B)
> The aim is to allow the servers to use both subnet A and subnet B. To
> allow this, any machine on both subnets must have source-specific
> routing configured, else packets originating from one ISP's AS will be
> directed at the other's router, and neither ISP cares for that.
> At the moment, I'm focusing on getting the second ISP properly added to
> the firewall box. The firewall box is using CentOS 6.4, and normally
> passes traffic back and forth via proxy_arp. None of my interfaces are
> NM_CONTROLLED, and NetworkManager is not installed, much less started.
> I've created a route-eth0:1 file that looks roughly like this:
> dev eth0:1 \
>   src \
>   from
> default via dev eth0:1 \
>   src \
>   from
> (Treat indented lines as continuations of the previous line)
> (No, the ISPs aren't giving me RFC1918 addresses; these are redacted.)
> If I run "ifup eth0:1", "ip route show" includes the lines:
> dev eth0  scope link  src
> dev eth0  proto kernel  scope link  src
> default via dev eth0
> Note that the "from" clause is missing. With the addition of
> a second default route on my firewall/gateway without any restriction on
> which traffic should go that way, my whole network, of course, tanks.
> I'm surprised it's been such a pain; I would have expected it to be a
> relatively common configuration. What's the proper way of doing
> source-specific routing on CentOS?
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos

Michael H. Warfield (AI4NB) | (770) 985-6132 |  mhw at WittsEnd.com
   /\/\|=mhw=|\/\/          | (678) 463-0932 |  http://www.wittsend.com/mhw/
   NIC whois: MHW9          | An optimist believes we live in the best of all
 PGP Key: 0x674627FF        | possible worlds.  A pessimist is sure of it!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: This is a digitally signed message part
URL: <http://lists.centos.org/pipermail/centos/attachments/20130501/3a240ea9/attachment-0005.sig>