On Wed, 24 Apr 2019 at 10:31, likun <kun.li at ucarinc.com> wrote: > Hello, Stephen, thank you for input. > > Yes, these servers have the same firewall rules, and both of them have the > same problem from time to time, most of time they are good. > > Actually, these servers are newly installed to be used as the Glusterfs > storage server, so not much data flowing at this time. > From the sysctl output, I suppose it can't be a conntrack table overflow : > > net.netfilter.nf_conntrack_count = 1116 > net.netfilter.nf_conntrack_max = 262144 > > And another tcpdump ouput of a successful ssh connection between these two > servers for reference: > > 21:41:53.225977 IP (tos 0x0, ttl 64, id 30083, offset 0, flags [DF], proto > TCP (6), length 60) > 10.3.3.3.49221 > 10.3.3.4.22: Flags [S], cksum 0x1ab0 (incorrect -> > 0x62bc), seq 320483660, win 29200, options [mss 1460,sackOK,TS val > 253838543 ecr 0,nop,wscale 7], length 0 > 21:41:53.226083 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP > (6), length 60) > 10.3.3.4.22 > 10.3.3.3.49221: Flags [S.], cksum 0x15da (correct), seq > 3543971827, ack 320483661, win 28960, options [mss 1460,sackOK,TS val > 94551278 ecr 253838543,nop,wscale 7], length 0 > 21:41:53.226116 IP (tos 0x0, ttl 64, id 30084, offset 0, flags [DF], proto > TCP (6), length 52) > 10.3.3.3.49221 > 10.3.3.4.22: Flags [.], cksum 0x1aa8 (incorrect -> > 0xb4e1), seq 1, ack 1, win 229, options [nop,nop,TS val 253838543 ecr > 94551278], length 0 > > I notice these have checksum's in them but the first set didn't. Was there an option changed in the tcpdump? It is clear that I am wrong about the [SA] being needed since this set does have an [S.] Sorry I don't have any clues.. as far as I can tell it should have worked. Could you put a log before the REJECT to see if something else stands out? > Likun > > -----邮件原件----- > 发件人: CentOS [mailto:centos-bounces at centos.org] 代表 Stephen John Smoogen > 发送时间: 2019年4月24日 18:35 > 收件人: CentOS mailing list > 主题: Re: [CentOS] Iptables blocks out going connetion some times > > On Wed, 24 Apr 2019 at 06:01, likun <kun.li at ucarinc.com> wrote: > > > Hi guys. > > > > There is a wierd problem with iptables recently, hopes somebody can > > help me. > > > > I have installed Centos 7.2.1511 on a bare metal Dell server these > > days, disabled firewalld and enabled iptables.services, and setup a > > group of very simple rules, as the following: > > > > > I believe you are saying both 10.3.3.3 and 10.3.3.4 have this same > firewall but I am not sure. > > > > > # iptables-save > > > > # Generated by iptables-save v1.4.21 on Tue Apr 23 09:15:14 2019 > > > > *filter > > > > :INPUT ACCEPT [0:0] > > > > :FORWARD ACCEPT [0:0] > > > > :OUTPUT ACCEPT [2449555:327804572] > > > > -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT > > > > -A INPUT -p icmp -j ACCEPT > > > > -A INPUT -i lo -j ACCEPT > > > > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 49152:49664 -m > > conntrack --ctstate NEW -j ACCEPT > > > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 49152:49664 -m conntrack > > --ctstate NEW -j ACCEPT > > > > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 24007 -m conntrack > > --ctstate NEW -j ACCEPT > > > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 24007 -m conntrack > > --ctstate NEW -j ACCEPT > > > > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 22 -m conntrack > > --ctstate NEW -j ACCEPT > > > > -A INPUT -s 10.3.7.0/25 -p tcp -m tcp --dport 22 -m conntrack > > --ctstate NEW -j ACCEPT > > > > -A INPUT -j REJECT --reject-with icmp-host-prohibited > > > > -A FORWARD -j REJECT --reject-with icmp-host-prohibited > > > > COMMIT > > > > # Completed on Tue Apr 23 09:15:14 2019 > > > > > > > > From time to time, when this server, say 10.3.3.3, trying to connect > > to port > > 24007 of another server 10.3.3.4, it will fail sometimes. from tcpdump > > output, you can see these packages: > > > > 22:49:05.992737 IP 10.3.3.3.49149 > 10.3.3.4.24007: Flags [S], seq > > 2454712274, win 29200, options [mss 1460,sackOK,TS val 24055648 ecr > > 0,nop,wscale 7], length 0 > > > > 22:49:05.992847 IP 10.3.3.4.24007 > 10.3.3.3.49149: Flags [S.], seq > > 3127562073, ack 2454712275, win 28960, options [mss 1460,sackOK,TS val > > 17803660 ecr 24055648,nop,wscale 7], length 0 > > > > 22:49:05.992872 IP 10.3.3.3 > 10.3.3.4: ICMP host 10.3.3.3 unreachable > > - admin prohibited, length 68 > > > > > > > So it looks like the packet getting rejected wasn't just a SYN packet but > had something extra tagged NONE tagged in it [S.]. Do the packets which > don't trigger an ICMP return contain a [SA] or a [S.] ? > > My guess is that something in that return packet is making the cstate not > be NEW or ESTABLISHED and so it is failing down to the drop. If the packets > which are accepted look the same then I would look to see if the conntrack > table is overflowing.. if the memory is tight and the traffic is high, > streams will get pushed out and you will get similar failures. That however > usally puts something in dmesg when it happens. Sorry I don't have a better > answer and I am looking forward to someone correcting my mistakes here :). > > > > > > > The package back from 10.3.3.4 is prohibited by 10.3.3.3, why? > > > > Obviously, the package that's been prohibited is the SYNC package that > > in response of the SYNC sent out by 10.3.3.3, it should match the > > iptables rule -m state --state RELATED,ESTABLISHED -j ACCEPT, but > > still be prohibited by the rule -j REJECT --reject-with > > icmp-host-prohibited. > > > > > > > > Also, the iptables -nvL output: > > > > > > > > # iptables -nvL > > > > Chain INPUT (policy ACCEPT 0 packets, 0 bytes) > > > > pkts bytes target prot opt in out source > > destination > > > > 8258 615K ACCEPT all -- * * 0.0.0.0/0 > > 0.0.0.0/0 state RELATED,ESTABLISHED > > > > 1 84 ACCEPT icmp -- * * 0.0.0.0/0 > > 0.0.0.0/0 > > > > 0 0 ACCEPT all -- lo * 0.0.0.0/0 > > 0.0.0.0/0 > > > > 0 0 ACCEPT tcp -- * * 172.22.0.0/16 > > 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW > > > > 0 0 ACCEPT tcp -- * * 10.3.3.0/25 > > 0.0.0.0/0 tcp dpts:49152:49664 ctstate NEW > > > > 0 0 ACCEPT tcp -- * * 172.22.0.0/16 > > 0.0.0.0/0 tcp dpt:24007 ctstate NEW > > > > 918 55080 ACCEPT tcp -- * * 10.3.3.0/25 > > 0.0.0.0/0 tcp dpt:24007 ctstate NEW > > > > 1 60 ACCEPT tcp -- * * 10.3.3.0/25 > > 0.0.0.0/0 tcp dpt:22 ctstate NEW > > > > 0 0 ACCEPT tcp -- * * 10.3.7.0/25 > > 0.0.0.0/0 tcp dpt:22 ctstate NEW > > > > 244 14640 REJECT all -- * * 0.0.0.0/0 > > 0.0.0.0/0 reject-with icmp-host-prohibited > > > > > > > > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) > > > > pkts bytes target prot opt in out source > > destination > > > > 0 0 REJECT all -- * * 0.0.0.0/0 > > 0.0.0.0/0 reject-with icmp-host-prohibited > > > > > > > > Chain OUTPUT (policy ACCEPT 7378 packets, 957K bytes) > > > > pkts bytes target prot opt in out source > > destination > > > > > > > > I noticed that the 0.0.0.0/0 0.0.0.0/0 reject-with > > icmp-host-prohibited one increase fast. > > > > > > > > Is this a bug of iptables, or is there something that I have not noticed? > > > > Any hints will be appreciated. > > > > Likun > > > > > > > > _______________________________________________ > > CentOS mailing list > > CentOS at centos.org > > https://lists.centos.org/mailman/listinfo/centos > > > > > -- > Stephen J Smoogen. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > https://lists.centos.org/mailman/listinfo/centos > -- Stephen J Smoogen.