Hello, Stephen, thank you for input.
Yes, these servers have the same firewall rules, and both of them have the same problem from time to time, most of time they are good.
Actually, these servers are newly installed to be used as the Glusterfs storage server, so not much data flowing at this time.
From the sysctl output, I suppose it can't be a conntrack table overflow :
net.netfilter.nf_conntrack_count = 1116 net.netfilter.nf_conntrack_max = 262144
And another tcpdump ouput of a successful ssh connection between these two servers for reference:
21:41:53.225977 IP (tos 0x0, ttl 64, id 30083, offset 0, flags [DF], proto TCP (6), length 60) 10.3.3.3.49221 > 10.3.3.4.22: Flags [S], cksum 0x1ab0 (incorrect -> 0x62bc), seq 320483660, win 29200, options [mss 1460,sackOK,TS val 253838543 ecr 0,nop,wscale 7], length 0 21:41:53.226083 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60) 10.3.3.4.22 > 10.3.3.3.49221: Flags [S.], cksum 0x15da (correct), seq 3543971827, ack 320483661, win 28960, options [mss 1460,sackOK,TS val 94551278 ecr 253838543,nop,wscale 7], length 0 21:41:53.226116 IP (tos 0x0, ttl 64, id 30084, offset 0, flags [DF], proto TCP (6), length 52) 10.3.3.3.49221 > 10.3.3.4.22: Flags [.], cksum 0x1aa8 (incorrect -> 0xb4e1), seq 1, ack 1, win 229, options [nop,nop,TS val 253838543 ecr 94551278], length 0
Likun
-----邮件原件----- 发件人: CentOS [mailto:centos-bounces@centos.org] 代表 Stephen John Smoogen 发送时间: 2019年4月24日 18:35 收件人: CentOS mailing list 主题: Re: [CentOS] Iptables blocks out going connetion some times
On Wed, 24 Apr 2019 at 06:01, likun kun.li@ucarinc.com wrote:
Hi guys.
There is a wierd problem with iptables recently, hopes somebody can help me.
I have installed Centos 7.2.1511 on a bare metal Dell server these days, disabled firewalld and enabled iptables.services, and setup a group of very simple rules, as the following:
I believe you are saying both 10.3.3.3 and 10.3.3.4 have this same firewall but I am not sure.
# iptables-save
# Generated by iptables-save v1.4.21 on Tue Apr 23 09:15:14 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [2449555:327804572]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 49152:49664 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 49152:49664 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 24007 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 24007 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -s 10.3.7.0/25 -p tcp -m tcp --dport 22 -m conntrack --ctstate NEW -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Tue Apr 23 09:15:14 2019
From time to time, when this server, say 10.3.3.3, trying to connect to port 24007 of another server 10.3.3.4, it will fail sometimes. from tcpdump output, you can see these packages:
22:49:05.992737 IP 10.3.3.3.49149 > 10.3.3.4.24007: Flags [S], seq 2454712274, win 29200, options [mss 1460,sackOK,TS val 24055648 ecr 0,nop,wscale 7], length 0
22:49:05.992847 IP 10.3.3.4.24007 > 10.3.3.3.49149: Flags [S.], seq 3127562073, ack 2454712275, win 28960, options [mss 1460,sackOK,TS val 17803660 ecr 24055648,nop,wscale 7], length 0
22:49:05.992872 IP 10.3.3.3 > 10.3.3.4: ICMP host 10.3.3.3 unreachable
- admin prohibited, length 68
So it looks like the packet getting rejected wasn't just a SYN packet but had something extra tagged NONE tagged in it [S.]. Do the packets which don't trigger an ICMP return contain a [SA] or a [S.] ?
My guess is that something in that return packet is making the cstate not be NEW or ESTABLISHED and so it is failing down to the drop. If the packets which are accepted look the same then I would look to see if the conntrack table is overflowing.. if the memory is tight and the traffic is high, streams will get pushed out and you will get similar failures. That however usally puts something in dmesg when it happens. Sorry I don't have a better answer and I am looking forward to someone correcting my mistakes here :).
The package back from 10.3.3.4 is prohibited by 10.3.3.3, why?
Obviously, the package that's been prohibited is the SYNC package that in response of the SYNC sent out by 10.3.3.3, it should match the iptables rule -m state --state RELATED,ESTABLISHED -j ACCEPT, but still be prohibited by the rule -j REJECT --reject-with icmp-host-prohibited.
Also, the iptables -nvL output:
# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
8258 615K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
1 84 ACCEPT icmp -- * * 0.0.0.0/00.0.0.0/0
0 0 ACCEPT all -- lo * 0.0.0.0/00.0.0.0/0
0 0 ACCEPT tcp -- * * 172.22.0.0/160.0.0.0/0 tcp dpts:49152:49664 ctstate NEW
0 0 ACCEPT tcp -- * * 10.3.3.0/250.0.0.0/0 tcp dpts:49152:49664 ctstate NEW
0 0 ACCEPT tcp -- * * 172.22.0.0/160.0.0.0/0 tcp dpt:24007 ctstate NEW
918 55080 ACCEPT tcp -- * * 10.3.3.0/25 0.0.0.0/0 tcp dpt:24007 ctstate NEW
1 60 ACCEPT tcp -- * * 10.3.3.0/250.0.0.0/0 tcp dpt:22 ctstate NEW
0 0 ACCEPT tcp -- * * 10.3.7.0/250.0.0.0/0 tcp dpt:22 ctstate NEW
244 14640 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 REJECT all -- * * 0.0.0.0/00.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT 7378 packets, 957K bytes)
pkts bytes target prot opt in out source destination
I noticed that the 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited one increase fast.
Is this a bug of iptables, or is there something that I have not noticed?
Any hints will be appreciated.
Likun
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-- Stephen J Smoogen. _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos