[CentOS] 答复: Iptables blocks out going connetion some times

Wed Apr 24 15:47:05 UTC 2019
Stephen John Smoogen <smooge at gmail.com>

On Wed, 24 Apr 2019 at 10:31, likun <kun.li at ucarinc.com> wrote:

> Hello, Stephen, thank you for input.
>
> Yes, these servers have the same firewall rules, and both of them have the
> same problem from time to time, most of time they are good.
>
> Actually, these servers are newly installed to be used as the Glusterfs
> storage server, so not much data flowing at this time.
> From the sysctl output, I suppose it can't be a conntrack table overflow :
>
> net.netfilter.nf_conntrack_count = 1116
> net.netfilter.nf_conntrack_max = 262144
>
> And another tcpdump ouput of a successful ssh connection between these two
> servers for reference:
>
> 21:41:53.225977 IP (tos 0x0, ttl 64, id 30083, offset 0, flags [DF], proto
> TCP (6), length 60)
>     10.3.3.3.49221 > 10.3.3.4.22: Flags [S], cksum 0x1ab0 (incorrect ->
> 0x62bc), seq 320483660, win 29200, options [mss 1460,sackOK,TS val
> 253838543 ecr 0,nop,wscale 7], length 0
> 21:41:53.226083 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP
> (6), length 60)
>     10.3.3.4.22 > 10.3.3.3.49221: Flags [S.], cksum 0x15da (correct), seq
> 3543971827, ack 320483661, win 28960, options [mss 1460,sackOK,TS val
> 94551278 ecr 253838543,nop,wscale 7], length 0
> 21:41:53.226116 IP (tos 0x0, ttl 64, id 30084, offset 0, flags [DF], proto
> TCP (6), length 52)
>     10.3.3.3.49221 > 10.3.3.4.22: Flags [.], cksum 0x1aa8 (incorrect ->
> 0xb4e1), seq 1, ack 1, win 229, options [nop,nop,TS val 253838543 ecr
> 94551278], length 0
>
>
I notice these have checksum's in them but the first set didn't. Was there
an option changed in the tcpdump?  It is clear that I am wrong about the
[SA] being needed since this set does have an [S.]  Sorry I don't have any
clues.. as far as I can tell it should have worked. Could you put a log
before the REJECT to see if something else stands out?



> Likun
>
> -----邮件原件-----
> 发件人: CentOS [mailto:centos-bounces at centos.org] 代表 Stephen John Smoogen
> 发送时间: 2019年4月24日 18:35
> 收件人: CentOS mailing list
> 主题: Re: [CentOS] Iptables blocks out going connetion some times
>
> On Wed, 24 Apr 2019 at 06:01, likun <kun.li at ucarinc.com> wrote:
>
> > Hi  guys.
> >
> > There is a wierd problem with iptables recently, hopes somebody can
> > help me.
> >
> > I have installed Centos 7.2.1511 on a bare metal Dell server these
> > days, disabled firewalld and enabled iptables.services, and setup a
> > group of very simple rules, as the following:
> >
> >
> I believe you are saying both 10.3.3.3 and 10.3.3.4 have this same
> firewall but I am not sure.
>
>
>
> > # iptables-save
> >
> > # Generated by iptables-save v1.4.21 on Tue Apr 23 09:15:14 2019
> >
> > *filter
> >
> > :INPUT ACCEPT [0:0]
> >
> > :FORWARD ACCEPT [0:0]
> >
> > :OUTPUT ACCEPT [2449555:327804572]
> >
> > -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
> >
> > -A INPUT -p icmp -j ACCEPT
> >
> > -A INPUT -i lo -j ACCEPT
> >
> > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 49152:49664 -m
> > conntrack --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 49152:49664 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 172.22.0.0/16 -p tcp -m tcp --dport 24007 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 24007 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.3.0/25 -p tcp -m tcp --dport 22 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -s 10.3.7.0/25 -p tcp -m tcp --dport 22 -m conntrack
> > --ctstate NEW -j ACCEPT
> >
> > -A INPUT -j REJECT --reject-with icmp-host-prohibited
> >
> > -A FORWARD -j REJECT --reject-with icmp-host-prohibited
> >
> > COMMIT
> >
> > # Completed on Tue Apr 23 09:15:14 2019
> >
> >
> >
> > From time to time, when this server, say 10.3.3.3, trying to connect
> > to port
> > 24007 of another server 10.3.3.4, it will fail sometimes. from tcpdump
> > output, you can see these packages:
> >
> > 22:49:05.992737 IP 10.3.3.3.49149 > 10.3.3.4.24007: Flags [S], seq
> > 2454712274, win 29200, options [mss 1460,sackOK,TS val 24055648 ecr
> > 0,nop,wscale 7], length 0
> >
> > 22:49:05.992847 IP 10.3.3.4.24007 > 10.3.3.3.49149: Flags [S.], seq
> > 3127562073, ack 2454712275, win 28960, options [mss 1460,sackOK,TS val
> > 17803660 ecr 24055648,nop,wscale 7], length 0
> >
> > 22:49:05.992872 IP 10.3.3.3 > 10.3.3.4: ICMP host 10.3.3.3 unreachable
> > - admin prohibited, length 68
> >
> >
> >
> So it looks like the packet getting rejected wasn't just a SYN packet but
> had something extra tagged NONE tagged in it [S.]. Do the packets which
> don't trigger an ICMP return contain a [SA] or a [S.] ?
>
> My guess is that something in that return packet is making the cstate not
> be NEW or ESTABLISHED and so it is failing down to the drop. If the packets
> which are accepted look the same then I would look to see if the conntrack
> table is overflowing.. if the memory is tight and the traffic is high,
> streams will get pushed out and you will get similar failures. That however
> usally puts something in dmesg when it happens. Sorry I don't have a better
> answer and I am looking forward to someone correcting my mistakes here :).
>
>
>
> >
> > The package back from 10.3.3.4 is prohibited by 10.3.3.3, why?
> >
> > Obviously, the package that's been prohibited is the SYNC package that
> > in response of the SYNC sent out by 10.3.3.3, it should match the
> > iptables rule -m state --state RELATED,ESTABLISHED -j ACCEPT, but
> > still be prohibited by the rule -j REJECT --reject-with
> > icmp-host-prohibited.
> >
> >
> >
> > Also, the iptables -nvL output:
> >
> >
> >
> > # iptables -nvL
> >
> > Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
> >
> > pkts bytes target     prot opt in     out     source
> > destination
> >
> >  8258  615K ACCEPT     all  --  *      *       0.0.0.0/0
> > 0.0.0.0/0            state RELATED,ESTABLISHED
> >
> >     1    84 ACCEPT     icmp --  *      *       0.0.0.0/0
> > 0.0.0.0/0
> >
> >     0     0 ACCEPT     all  --  lo     *       0.0.0.0/0
> > 0.0.0.0/0
> >
> >     0     0 ACCEPT     tcp  --  *      *       172.22.0.0/16
> > 0.0.0.0/0            tcp dpts:49152:49664 ctstate NEW
> >
> >     0     0 ACCEPT     tcp  --  *      *       10.3.3.0/25
> > 0.0.0.0/0            tcp dpts:49152:49664 ctstate NEW
> >
> >     0     0 ACCEPT     tcp  --  *      *       172.22.0.0/16
> > 0.0.0.0/0            tcp dpt:24007 ctstate NEW
> >
> >   918 55080 ACCEPT     tcp  --  *      *       10.3.3.0/25
> > 0.0.0.0/0            tcp dpt:24007 ctstate NEW
> >
> >     1    60 ACCEPT     tcp  --  *      *       10.3.3.0/25
> > 0.0.0.0/0            tcp dpt:22 ctstate NEW
> >
> >     0     0 ACCEPT     tcp  --  *      *       10.3.7.0/25
> > 0.0.0.0/0            tcp dpt:22 ctstate NEW
> >
> >   244 14640 REJECT     all  --  *      *       0.0.0.0/0
> > 0.0.0.0/0            reject-with icmp-host-prohibited
> >
> >
> >
> > Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
> >
> > pkts bytes target     prot opt in     out     source
> > destination
> >
> >     0     0 REJECT     all  --  *      *       0.0.0.0/0
> > 0.0.0.0/0            reject-with icmp-host-prohibited
> >
> >
> >
> > Chain OUTPUT (policy ACCEPT 7378 packets, 957K bytes)
> >
> > pkts bytes target     prot opt in     out     source
> > destination
> >
> >
> >
> > I noticed that the 0.0.0.0/0 0.0.0.0/0 reject-with
> > icmp-host-prohibited one increase fast.
> >
> >
> >
> > Is this a bug of iptables, or is there something that I have not noticed?
> >
> > Any hints will be appreciated.
> >
> > Likun
> >
> >
> >
> > _______________________________________________
> > CentOS mailing list
> > CentOS at centos.org
> > https://lists.centos.org/mailman/listinfo/centos
> >
>
>
> --
> Stephen J Smoogen.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>


-- 
Stephen J Smoogen.