Thanks for your responses. I have tried some of the configurations that you have mentioned but not all of them. I will try the others tonight when the server is less active. In the meantime I will read the links you sent.

Thanks again. This is starting to drive me crazy.

On Wed, Jul 1, 2009 at 3:07 PM, Alexander Dalloz <ad+lists@uni-x.orgad%2Blists@uni-x.org

wrote:

Doug Coats schrieb:

...

I have a server with 4 nics. Two are using different internet

connections,

...

both with static IP's, and two are connected to our private network.

The two internet facing nics seem to be battling over the gateway designation. Which ever I designate as the gateway the other stops responding to incoming traffic. I need both to listen to inbound

traffic.

...

One for our main web page and the other for another web server. I found some information of Iproute2 but that did not seem to solve my issues.

Here are the config files:

ifcfg-eth2 DEVICE=eth2 BOOTPROTO=static HWADDR=00:24:E8:52:92:8E ONBOOT=yes IPADDR=xx.yy.51.45 NETMASK=255.255.255.252 GATEWAY=xx.yy.51.46

ifcfg-eth3 DEVICE=eth3 BOOTPROTO=static HWADDR=00:24:E8:52:92:90 ONBOOT=yes IPADDR=aa.bb.166.2 NETMASK=255.255.255.224 GATEWAY=aa.bb.166.1

network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=heritage01 GATEWAY=xx.yy.51.46 GATEWAYDEV=eth2

rt_tables # # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 200 Cable 201 T1 iproute2 routes ip route add xx.yy.51.44/30 dev eth2 src xx.yy.51.46 table Cable ip route add default via xx.yy.51.46 ip route add xx.yy.51.44/30 dev eth2 src xx.yy.51.46 ip route add default via xx.yy.51.46 ip rule add from xx.yy.51.46 table Cable ip route add aa.bb.166.0/27 dev eth3 src aa.bb.165.2 table T1 ip route add default via aa.bb.166.1 ip route add aa.bb.166.0/27 dev eth3 src aa.bb.166.2 ip rule add from aa.bb.166.2 table T1 ip route flush cache

Other people seem to be using this configuration successfully but I seem

to

...

be missing something important.

Any help is greatly appreciated!!!!!!! Thanks!

The GATEWAY parameter within the ifcfg-<ethX> configuration files tries to set the default gateway. There can be just 1 default gateway for a system. So, the interface which comes up first will set the route and the other has no effect.

Remove the GATEWAY entry from all ifcfg-<ethX> files and place the GATEWAY parameter into /etc/sysconfig/network only.

Running different networks you must can set additional routings by route-<ethX> configuration files inside /etc/sysconfig/network-scripts/.

Given your default gateway is set as xx.yy.51.46 for the network configured on eth2, you set to use a gateway aa.bb.166.1 for other destinations by

ip route add targetnet/CIDR via aa.bb.166.1 dev eth3

Please see

http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s1-networkscripts-sta...

You multiple uplink topic is documented in this howto

http://lartc.org/howto/lartc.rpdb.multiple-links.html

Alexander

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

I changed those two rules to no affect. I have also done the above mentioned things. My tcpdump shows connections comming in on eth3 but there is no outbound communication.

Example:

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth3, link-type EN10MB (Ethernet), capture size 96 bytes 20:21:11.915624 IP 41.178.204.108.25591 > ipaa-bb-166-2.z166-152-67.customer.algx.net.smtp: S 3528290937:3528290937(0) win 65535 <mss 1442,nop,wscale 2,nop,nop,sackOK> 20:21:14.892106 IP 41.178.204.108.25591 > ipaa-bb-166-2.z166-152-67.customer.algx.net.smtp: S 3528290937:3528290937(0) win 65535 <mss 1442,nop,wscale 2,nop,nop,sackOK> 20:21:16.017705 20:21:16.663301 IP newsletter.publicradio.org.54562 > ipaa-bb-166-2.z166-152-67.customer.algx.net.smtp: S 1665702165:1665702165(0) win 5840 <mss 1460,sackOK,timestamp 3732071063 0,nop,wscale 3> 20:21:17.658835 IP client-201.230.112.209.speedy.net.pe.12095 > ipaa-bb-166-2.z166-152-67.customer.algx.net.smtp: S 2778637809:2778637809(0) win 65535 <mss 1400,nop,wscale 1,nop,nop,sackOK>

An abreviated lsof -i4 -n

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME mysqld 4004 mysql 10u IPv4 8750 TCP *:mysql (LISTEN) spamd 4056 root 5u IPv4 8943 TCP 127.0.0.1:783 (LISTEN) sendmail 4076 root 4u IPv4 9094 TCP *:smtp (LISTEN) httpd 4110 root 4u IPv4 9117 TCP aa.bb.166.2:http (LISTEN) httpd 4166 apache 4u IPv4 9117 TCP aa.bb.166.2:http (LISTEN) httpd 4167 apache 4u IPv4 9117 TCP aa.bb.166.2:http (LISTEN) httpd 4168 apache 4u IPv4 9117 TCP aa.bb.166.2:http (LISTEN) So it appears to me that the device is listening but it never picks up the phone.

Any thoughts?

On Wed, Jul 1, 2009 at 7:00 PM, Christopher Chan < christopher.chan@bradbury.edu.hk> wrote:

...

ip rule add from xx.yy.51.46 table Cable ip rule add from aa.bb.166.2 table T1

Other people seem to be using this configuration successfully but I seem to be missing something important.

Change the above rules to:

ip rule add from xx.yy.51.46 to default lookup Cable

ip rule add from aa.bb.166.2 to default lookup T1 _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Thanks for your responce!

If I open up Iptables to accept everything on eth3 and eth2 or turn it off all together I get the same results.

I will have to look through your iproute2 commands to see how they match up to mine. Do you see anything different in mine that would indicate the issue?

On Wed, Jul 1, 2009 at 9:25 PM, Christopher Chan < christopher.chan@bradbury.edu.hk> wrote:

Hello Doug,

Here are the iproute2 commands for a multigateway setup I did.

ip route add 220.232.217.72/29 dev eth0 proto kernel scope link src 220.232.217.78 table pacific ip route add 10.9.0.0/17 dev eth1 proto kernel scope link src 10.9.1.101 table pacific ip route add default via 220.232.217.73 dev eth0 table pacific ip route add 203.174.45.24/29 dev eth2 proto kernel scope link src 203.174.45.30 table wharf ip route add 10.9.0.0/17 dev eth1 proto kernel scope link src 10.9.1.101 table wharf ip route add default via 203.174.45.25 dev eth2 table wharf ip route add 59.152.193.16/29 dev eth2 proto kernel scope link src 59.152.193.22 table frahw ip route add 10.9.0.0/17 dev eth1 proto kernel scope link src 10.9.1.101 table frahw ip route add default via 59.152.193.17 dev eth2 table wharf ip route add 10.9.0.0/17 dev eth1 proto kernel scope link src 10.9.1.101 table saurer ip route add 10.0.0.0/8 via 10.9.0.1 dev eth1 table saurer

ip rule add from 220.232.217.78 to default lookup pacific ip rule add from 203.174.45.30 to default lookup wharf

You may also want to check your firewall rules too.

I must also say that there are no GATEWAY entries in my ifcfg-ethx. Only one in network. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Doug Coats wrote:

I am not understanding somethings very fundamental to me troubleshooting this issue.

How do I clear the iproute2's that I have created?

They are cleared by being deleted or a reboot takes place.

Try 'ip route help' and 'ip rule help'

How do I tell if they are in affect?

Now that is a good one...ping/telnet tests from another host...

What should they look like when listed if they are configured properly?

Here is my output from ip rule list

0: from all lookup local 32764: from 203.174.45.30 lookup wharf 32765: from 220.232.217.78 lookup pacific 32766: from all lookup main 32767: from all lookup default

Lower numbers mean higher priority.

Routing tables:

ip route list table wharf: 203.174.45.24/29 dev eth2 proto kernel scope link src 203.174.45.30 10.9.0.0/17 dev eth1 proto kernel scope link src 10.9.1.101 default via 203.174.45.25 dev eth2

ip route list table pacific 220.232.217.72/29 dev eth0 proto kernel scope link src 220.232.217.78 10.9.0.0/17 dev eth1 proto kernel scope link src 10.9.1.101 default via 220.232.217.73 dev eth0

ip route list will give you your main routing table which is the same one that route -n will report

I am convinced that my iproute2 tables are all messed up....In fact I found some errors in my initial post.

# ip rule list 0: from all lookup 255 32766: from all lookup main 32767: from all lookup default

/etc/iproute2/rt_tables # # reserved values # 255 local 254 main 253 default 0 unspec # # local # #1 inr.ruhep 200 Cable 201 T1 So Cable and T1 are not even showing up in the list.

When I run my ip commands from the command line I get the following results for each line.

# ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 table Cable # ip route add default via 173.11.51.45 table Cable # ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 RTNETLINK answers: File exists # ip route add default via 173.11.51.46 RTNETLINK answers: File Exists # ip rule add from 173.11.51.45 to default lookup Cable RTNETLINK answers: Numerical result out of range

# ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 table T1 # ip route add default via 67.152.166.2 table T1 # ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 RTNETLINK answers: File Exists # ip rule add from 67.152.166.2 to default lookup T1 RTNETLINK answers: Numerical result out of range So the first few lines run fine for each table and then I start getting errors. Are these all the commands that I need to use to set this up? I know that when I first entered them by hand the T1 and Cable showed up in the tables list. Am I going crazy?

So if I run the following two commands the rules show up in the list.

ip rule add from 173.11.51.46 table Cable ip rule add from 67.152.166.2 table T1

# ip rule list 0: from all lookup 255 32764: from 67.152.166.2 lookup T1 32765: from 173.11.51.46 lookup Cable 32766: from all lookup main 32767: from all lookup default

So why does that work but my other command not?

Doug Coats wrote:

So if I run the following two commands the rules show up in the list.

ip rule add from 173.11.51.46 table Cable ip rule add from 67.152.166.2 table T1

# ip rule list 0: from all lookup 255 32764: from 67.152.166.2 lookup T1 32765: from 173.11.51.46 lookup Cable 32766: from all lookup main 32767: from all lookup default

So why does that work but my other command not?

Which other commands? The ip route ones?

ip route list table T1 ip route list table Cable

I am ecstatically confused.

After I entered the last two commands my routing is working the way that I need it to.

ip rule add from 173.11.51.46 table Cable ip rule add from 67.152.166.2 table T1

The problem is I don't know which actual commands worked. I had just rebooted. The other ip commands were all in the rc.local file so they all ran.

Why did the above commands make the Cable and T1 rules show up in the rules list but the following two did not?

ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 table Cable ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 table T1 Do I need all of the commands? So that what is in rc.local looks like this?

ip rule add from 173.11.51.46 table Cable ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 table Cable ip route add default via 173.11.51.45 table Cable ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 ip route add default via 173.11.51.46 ip rule add from 173.11.51.45 to default lookup Cable ip rule add from 67.152.166.2 table T1 ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 table T1 ip route add default via 67.152.166.2 table T1 ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 ip rule add from 67.152.166.2 to default lookup T1 ip route flush cache

Doug Coats wrote:

I am ecstatically confused.

After I entered the last two commands my routing is working the way that I need it to.

ip rule add from 173.11.51.46 table Cable ip rule add from 67.152.166.2 table T1

The problem is I don't know which actual commands worked. I had just rebooted. The other ip commands were all in the rc.local file so they all ran.

Why did the above commands make the Cable and T1 rules show up in the rules list but the following two did not?

The following too are not ip rules. They are ip routing table entries. They will show up if you run 'ip route list table Cable'

ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 table Cable ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 table T1 Do I need all of the commands? So that what is in rc.local looks like this?

Take a look at the ones I posted...they are taken from rc.local since I have not taken to effort to integrate them elsewhere per interface.

ip rule add from 173.11.51.46 table Cable ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 table Cable ip route add default via 173.11.51.45 table Cable ip route add 173.11.51.44/30 dev eth2 src 173.11.51.45 ip route add default via 173.11.51.46 ip rule add from 173.11.51.45 to default lookup Cable ip rule add from 67.152.166.2 table T1 ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 table T1 ip route add default via 67.152.166.2 table T1 ip route add 67.152.166.0/27 dev eth3 src 67.152.166.2 ip rule add from 67.152.166.2 to default lookup T1 ip route flush cache

I find it interesting that you have ip rule commands before the relevant ip route commands for the custom routing table...

As a follow up issue. The only thing that is not working properly is that I can not pull up my website that is hosted on this server from our private network.

Do I need iproutes for my other two nics? I have never needed them before.

That is because you never redirected routing lookups to the custom tables. You can either add routing entries for your internal network into those two custom tables or you can add two SNAT rules assuming you also use the box as a nat box for the Internet.

Doug Coats wrote:

Christopher - you have been a great help!

My internal network ip is 192.168.4.1 and I need it to access the aa.bb.166.2 interface or eth3. What would the rule look like that I need to add?

ip route add 192.168.4.0/24 (??? i don't know your subnet - guessing your netmask here) dev ethx (replace x with the 192.168.4.1 interface) proto kernel scope link src 192.168.4.1 table Cable ip route add 192.168.4.0/24 (??? i don't know your subnet - guessing your netmask here) dev ethx (replace x with the 192.168.4.1 interface) proto kernel scope link src 192.168.4.1 table T1

That should fix things for both interfaces. Or use the command only for the table that corresponds to eth3.

cheers,

Christopher

On Thu, Jul 2, 2009 at 6:51 PM, Christopher Chan <christopher.chan@bradbury.edu.hk mailto:christopher.chan@bradbury.edu.hk> wrote:

> As a follow up issue.  The only thing that is not working
properly is
> that I can not pull up my website that is hosted on this server from
> our private network.
>
> Do I need iproutes for my other two nics?  I have never needed them
> before.
>
>

That is because you never redirected routing lookups to the custom
tables. You can either add routing entries for your internal network
into those two custom tables or you can add two SNAT rules
assuming you
also use the box as a nat box for the Internet.
_______________________________________________
CentOS mailing list
CentOS@centos.org <mailto:CentOS@centos.org>
http://lists.centos.org/mailman/listinfo/centos

---

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

On Thu, Jul 2, 2009 at 10:05 AM, Lorenzo Quatrinilorenzo.quatrini@gmail.com wrote:

John R Pierce ha scritto:

...

now, these commands are NOT persistent, and, AFAIK, RHEL has no provision for `ip route` or rule commands, so I end up sticking this stuff in /etc/rc.d/rc.local or something.

I was facing the same problem some time ago... what is the right place to put "ip route" commands and configuration files? Does someone have a better place other than rc.local? If I put them in rc.local, is there a way to issue a command like "service network restart"?

TIA

Lorenzo Quatrini _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Yeah, in /etc/sysconfig/network-scripts/ e.g. /etc/sysconfig/network-scripts/route-eth0

I got something like: aaa.bbb.ccc.0/24 dev eth0 in there

Lorenzo Quatrini wrote:

John R Pierce ha scritto:

...

now, these commands are NOT persistent, and, AFAIK, RHEL has no provision for `ip route` or rule commands, so I end up sticking this stuff in /etc/rc.d/rc.local or something.

I was facing the same problem some time ago... what is the right place to put "ip route" commands and configuration files? Does someone have a better place other than rc.local?

Yes, you can create rule-interfacename file under /etc/sysconfig/network-scripts directory for CentOS 5.x

If I put them in rc.local, is there a way to issue a command like "service network restart"?

TIA

Lorenzo Quatrini _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Lorenzo Quatrini wrote:

More detail on this: digging a bit on the network-scripts I found that there are few checks and calls to /sbin/ifup-pre-local, /sbin/ifup-local, /sbin/ifdown-pre-local, /sbin/ifdown-local, so I guess that this is the "right" place to put ip routes and rules commands.

i dunno, i would sort of assume the -local functions are for the localhost interface (/dev/lo), and the idea of having system specific config files in /sbin/ is somewhat abhorrant.

in fact, upon some digging, it looks like you put RULES in /etc/sysconfig/network-scripts/rule-ethX and ROUTES in /etc/sysconfig/network-scripts/route-ethX

the lines in the rule-* file are run prefixed by `ip rule add` while the lines in the route-* file are prefixed by `ip route add` on an interface 'up' event, and on the corresponding 'down' event, they are prefixed by ip {route|rule} del ....

so in fact, my 'AFAIK there is no provision' is in fact wrong, at least in CentOS 5.x....

...

in fact, upon some digging, it looks like you put RULES in /etc/sysconfig/network-scripts/rule-ethX and ROUTES in /etc/sysconfig/network-scripts/route-ethX

the lines in the rule-* file are run prefixed by `ip rule add` while the lines in the route-* file are prefixed by `ip route add` on an interface 'up' event, and on the corresponding 'down' event, they are prefixed by ip {route|rule} del ....

So does anyone have a clue on which is the syntax for creating complex routes and rules using route-* and rule-*?

Heh. I have not had time to go through the network scripts...last thing I heard about was route-* but that was Centos/RHEL 4. Docs have been lacking on this side of things...

On Fri, Jul 3, 2009 at 1:40 PM, Doug Coats dcoatshca@gmail.com wrote:

Adding the routes and rules to the appropriate files in network-scripts did not work. It gave me a number of errors and did not create the rules after reboot or/and network restart.

You will have to show the file contents and the actual error messages to get any serious help.

"did not work" is not enough info.

I have used route-* on CentOS 5. It can work.

-Bob

Bob you are so right!

I made the mistake of not removing "ip rule add" and "ip route add" from each of the lines.

Which caused the script, which wanted to add those itself, get confused.

Now I have removed those and it brings up my routes on reboot and network restart.

You will have to show the file contents and the actual error messages to

get >any serious help.

"did not work" is not enough info.

Now where would the proper place be to put a route for load balancing like:

ip route add default scope global nexthop via xx.yy.51.46 dev eth2 weight 3 nexthop via aa.bb.166.2 dev eth3 weight 1

Hey Doug,

Congrats! Now that we have helped you, it is your turn to help us! :-D

Let us know when you get that one working!

Cheers,

Christopher