Hi folks,
I have 2 firewalls, setup with Centos 5.2. They are also routers, connected to 2 upstream routers.
I have some cases where connections from servers to the internet leave my network via router2 and answers come back via router1. So I added conntrack tools to both routers/firewalls to synchronize the session tables (using ftfw procotol).
That works as expected. If e.g. I ping from an inside server to somewhere outside, ICMP request leaves via router2, the answer comes back via router1. conntrack -e on router1 shows this session (as unreplied), BUT the firewall blocks it as new connection - that means iptables does not recognize conntrackd's addition to the session table.
Seems that I have a conceptional misunderstanding here - but I do not find anything that could be wrong. Could somebody please help? I am stuck.
Any hint or help is appreciated.
Dirk
On Sunday 10 August 2008 08:36, Dirk H. Schulz wrote:
That works as expected. If e.g. I ping from an inside server to somewhere outside, ICMP request leaves via router2, the answer comes back via router1. conntrack -e on router1 shows this session (as unreplied), BUT the firewall blocks it as new connection - that means iptables does not recognize conntrackd's addition to the session table.
First off if you have traffic leaving one router and coming back on another router that is Asynchronous routing and is not a good thing, as you are seeing.
Firewall 1 doesn't know what firewall 2 is doing so firewall 1 is going to block this traffic as it was setup to do. Firewall 1 is thinking this is a new connection.
Since I don't know your setup my question is;
1. how many Internet connections do you have? 2. does router 2 have a valid public ip on the interface connecting to the Internet?
Hi Robert,
--On 10. August 2008 10:04:37 -0400 Robert Spangler mlists@zoominternet.net wrote:
On Sunday 10 August 2008 08:36, Dirk H. Schulz wrote:
That works as expected. If e.g. I ping from an inside server to somewhere outside, ICMP request leaves via router2, the answer comes back via router1. conntrack -e on router1 shows this session (as unreplied), BUT the firewall blocks it as new connection - that means iptables does not recognize conntrackd's addition to the session table.
First off if you have traffic leaving one router and coming back on another router that is Asynchronous routing and is not a good thing, as you are seeing.
Firewall 1 doesn't know what firewall 2 is doing so firewall 1 is going to block this traffic as it was setup to do. Firewall 1 is thinking this is a new connection.
That is why I used conntrack-tools to synchronize the session tables of both firewalls. According to "conntrackd -e" it works - it shows (e. g. on router 1) the sessions that have been synchronized over (e.g. from router 2).
But the sync'd sessions seem not to bother netfilter.
Since I don't know your setup my question is;
- how many Internet connections do you have?
This is still in setup phase, but they will be very many.
- does router 2 have a valid public ip on the interface connecting to
the Internet?
Yes. Both routers have public ips as they both are connected to upstream routers.
Dirk
On Sunday 10 August 2008 11:03, Dirk H. Schulz wrote:
That works as expected. If e.g. I ping from an inside server to somewhere outside, ICMP request leaves via router2, the answer comes back via router1. conntrack -e on router1 shows this session (as unreplied), BUT the firewall blocks it as new connection - that means iptables does not recognize conntrackd's addition to the session table.
First off if you have traffic leaving one router and coming back on another router that is Asynchronous routing and is not a good thing, as you are seeing.
Firewall 1 doesn't know what firewall 2 is doing so firewall 1 is going to block this traffic as it was setup to do. Firewall 1 is thinking this is a new connection.
That is why I used conntrack-tools to synchronize the session tables of both firewalls. According to "conntrackd -e" it works - it shows (e. g. on router 1) the sessions that have been synchronized over (e.g. from router 2).
But the sync'd sessions seem not to bother netfilter.
Since I don't know your setup my question is;
- how many Internet connections do you have?
This is still in setup phase, but they will be very many.
- does router 2 have a valid public ip on the interface connecting to
the Internet?
Yes. Both routers have public ips as they both are connected to upstream routers.
OK, I don't know this tool you are using to syn the conntracking of all the firewalls. Could you post a link to it?
Now for the fun stuff. Why would you have many Internet connection that do not return the same path they go out on? sounds like you really only have one true connection with one true IP to the Internet. That would explain why traffic leaving on interface 2 comes back on interface 1.
Without knowing your setup I'm not going to guess at this.
Hi Robert,
--On 10. August 2008 13:56:22 -0400 Robert Spangler mlists@zoominternet.net wrote:
- snip -
OK, I don't know this tool you are using to syn the conntracking of all the firewalls. Could you post a link to it?
Yes, of course: http://www.netfilter.org/projects/conntrack-tools/index.html
Now for the fun stuff. Why would you have many Internet connection that do not return the same path they go out on? sounds like you really only have one true connection with one true IP to the Internet. That would explain why traffic leaving on interface 2 comes back on interface 1.
It is two routers that are connected to 2 upstream routers; all four use OSPFv2 for routing between them. I have not finetuned OSPF so far to avoid asynchronous routing - I want to to do the connection table synchronization stuff before because I have to do it anyway (in case of a router crash) and now I have an ideal testbed (because of the asynchronous routing).
Without knowing your setup I'm not going to guess at this.
The setup is as follows: Every Router has - an external interface with public ip address each resting in a small separate subnet that connects to the upstream router - an interface for inter router connections (private ip addresses) - 2 additional interfaces to server LANs - both routers have an interface to both of the 2 server LANs both server LAN interface use shared virtual ips additionally
If you need more detailed information I could offer the OSFP configuration (XORP).
Here is the configuration for conntrackd (I have omitted buffer sizes etc.):
Sync { Mode FTFW { ResendBufferSize 262144 CommitTimeout 180 ACKWindowSize 20 } Multicast { IPv4_address 225.0.0.50 IPv4_interface 192.168.11.1 Interface eth1 Group 3780 } Checksum on CacheWriteThrough On } General { HashSize 8192 HashLimit 65535
- snip -
IgnoreTrafficFor { IPv4_address INTER_ROUTER_INTERFACE IPv4_address EXTERNAL_INTERFACE IPv4_address INTERNAL_INTERFACE1 IPv4_address INTERNAL_VIRTUAL_IP IPv4_address INTERNAL_INTERFACE2 }
IgnoreProtocol { IGMP VRRP }
The setup works - using "conntrackd -e" I can see the connection table entries the other router's conntrackd has synchronized. What I cannot check is if the receiving conntrackd writes the received entries into the kernels connection tracking table. Example:
udp 17 30 src=124.165.230.206 dst=93.94.81.82 sport=2040 dport=1434
[UNREPLIED] [active since 6s]
tcp 6 120 SYN_SENT src=93.185.115.91 dst=93.94.80.133 sport=4290
dport=135 [UNREPLIED] [active since 46s]
So I hope to find someone on the list have done this kind of setup before.
Thanks for your interest so far.
Dirk
On Sun, 2008-08-10 at 20:28 +0200, Dirk H. Schulz wrote:
Hi Robert,
--On 10. August 2008 13:56:22 -0400 Robert Spangler mlists@zoominternet.net wrote:
- snip -
OK, I don't know this tool you are using to syn the conntracking of all the firewalls. Could you post a link to it?
Yes, of course: http://www.netfilter.org/projects/conntrack-tools/index.html
Now for the fun stuff. Why would you have many Internet connection that do not return the same path they go out on? sounds like you really only have one true connection with one true IP to the Internet. That would explain why traffic leaving on interface 2 comes back on interface 1.
It is two routers that are connected to 2 upstream routers; all four use OSPFv2 for routing between them. I have not finetuned OSPF so far to avoid asynchronous routing - I want to to do the connection table synchronization stuff before because I have to do it anyway (in case of a router crash) and now I have an ideal testbed (because of the asynchronous routing).
Without knowing your setup I'm not going to guess at this.
The setup is as follows: Every Router has
- an external interface with public ip address each resting in a small
separate subnet that connects to the upstream router
- an interface for inter router connections (private ip addresses)
- 2 additional interfaces to server LANs - both routers have an interface
to both of the 2 server LANs both server LAN interface use shared virtual ips additionally
If you need more detailed information I could offer the OSFP configuration (XORP).
Here is the configuration for conntrackd (I have omitted buffer sizes etc.):
Sync { Mode FTFW { ResendBufferSize 262144 CommitTimeout 180 ACKWindowSize 20 } Multicast { IPv4_address 225.0.0.50 IPv4_interface 192.168.11.1 Interface eth1 Group 3780 } Checksum on CacheWriteThrough On } General { HashSize 8192 HashLimit 65535
- snip -
IgnoreTrafficFor { IPv4_address INTER_ROUTER_INTERFACE IPv4_address EXTERNAL_INTERFACE IPv4_address INTERNAL_INTERFACE1 IPv4_address INTERNAL_VIRTUAL_IP IPv4_address INTERNAL_INTERFACE2 }
IgnoreProtocol { IGMP VRRP }
The setup works - using "conntrackd -e" I can see the connection table entries the other router's conntrackd has synchronized. What I cannot check is if the receiving conntrackd writes the received entries into the kernels connection tracking table.
yum install iptstate iptstate Also: cat /proc/net/nf_conntrack
The doc says you must have kernel 2.6.18 or later. It looks like there are some iptables features that you can use that will not allow this to work. Are you in compliance with all of the dependencies listed in http://conntrack-tools.netfilter.org/conntrackd.html ?
Nataraj
Example:
udp 17 30 src=124.165.230.206 dst=93.94.81.82 sport=2040 dport=1434
[UNREPLIED] [active since 6s]
tcp 6 120 SYN_SENT src=93.185.115.91 dst=93.94.80.133 sport=4290
dport=135 [UNREPLIED] [active since 46s]
So I hope to find someone on the list have done this kind of setup before.
Hello Nataraj,
--On 12. August 2008 22:56:48 -0700 Nataraj incoming-centos@rjl.com wrote:
On Sun, 2008-08-10 at 20:28 +0200, Dirk H. Schulz wrote:
- snip -
The setup works - using "conntrackd -e" I can see the connection table entries the other router's conntrackd has synchronized. What I cannot check is if the receiving conntrackd writes the received entries into the kernels connection tracking table.
- snip -
Also: cat /proc/net/nf_conntrack
Okay, that was good (it is ip_conntrack, but never mind). Now I now that the kernel connection table does NOT get updated. Just have to find out why.
The doc says you must have kernel 2.6.18 or later. It looks like there are some iptables features that you can use that will not allow this to work. Are you in compliance with all of the dependencies listed in http://conntrack-tools.netfilter.org/conntrackd.html ?
Yes, the libraries are installed. The kernel should meet the prerequisites: CONFIG_NF_CONNTRACK=m: yes CONFIG_NF_CONNTRACK_IPV4=m: no, did not find it, could not enable it CONFIG_NETFILTER_NETLINK=m: yes, CONFIG_NF_CT_NETLINK=m: yes, it is called NF_CONNTRACK_NETLINK=m CONFIG_NF_CONNTRACK_EVENTS=y: yes
So only CONFIG_NF_CONNTRACK_IPV4 module is missing, but I thought that connection tracking would not work at all (even on just one netfilter instance) if a dedicated module für IPv4 additionally to the general NF_CONNTRACK module would really be needed.
Is there a debug mode for conntrackd where I can get more verbose logging to find out why conntrackd does not update the kernel connection table? Docs do not mention a debug mode, but maybe ...
By the way, when committing manually (conntrackd -c) I get the following entries in the log:
[Tue Aug 12 12:51:49 2008] (pid=22668) [notice] Committed 139 new entries [Tue Aug 12 12:51:49 2008] (pid=22668) [notice] 2 entries can't be
committed
[Tue Aug 12 12:51:54 2008] (pid=22671) [notice] committing external cache [Tue Aug 12 12:51:54 2008] (pid=22671) [ERROR] commit: Invalid argument Tue Aug 12 12:51:54 2008 tcp 6 180 SYN_SENT src=88.217.141.81
dst=93.94.80.2 sport=54930 dport=22 [UNREPLIED] src=93.94.80.2 dst=88.217.141.81 sport=22 dport=54930
[Tue Aug 12 12:51:54 2008] (pid=22671) [ERROR] commit: Invalid argument Tue Aug 12 12:51:54 2008 tcp 6 180 SYN_SENT src=88.217.141.81
dst=93.94.80.2 sport=54929 dport=22 [UNREPLIED] src=93.94.80.2 dst=88.217.141.81 sport=22 dport=54929
[Tue Aug 12 12:51:54 2008] (pid=22671) [notice] Committed 139 new entries
[Tue Aug 12 12:51:54 2008] (pid=22671) [notice] 2 entries can't be committed
Why can not all cache entries be committed? I did not find much about this. My kernel is a 2.6.18-92.1.6.el5 (CentOS 5).
Thanks for your help.
Dirk
-
Dirk H. Schulz -
Nataraj -
Robert Spangler