Last week, I started seeing very strange behavior in one of the networks that I manage.
The office LAN uses a Linux firewall which masquerades their workstations over their DSL connection. There are probably ~75 workstations in the office LAN.
Their mail server is in a collocated facility nearby. That server has an RFC1918 address; its router does SNAT to forward packets to the system.
Both the office firewall and the mail server are currently running fully patched CentOS 5.2.
Here's the weird part: If a machine running Linux in the office lan attempts to connect to the mail server on any TCP port, there's a small chance that the server will simply ignore the SYN packets. It doesn't log any errors. If I'm running tcpdump, I see the incoming SYN packets, but no reply. If I use iptables to log the packets, information about the packet is saved in the messages file. If I capture the packets and use wireshark to analyze them, I don't see anything odd: the checksums are good and I can't see any difference between a SYN packet that gets a SYN+ACK and one that's ignored (beyond the obvious: different timestamps and checksums). The problem doesn't seem to affect Windows workstations in the office LAN. As far as I can tell, only SYN packets are dropped. I don't see delays in established connections.
I've attached a file that contains, first, the output of tcpdump which shows packets to or from the office's firewall address, as recorded by the destination server. The first four SYN packets are ignored, but the kernel proceeds with the TCP handshake after the fifth SYN packet.
Second, the file contains the log messages which are recorded as a result of these iptables rules:
iptables -A INPUT -p tcp -s officefw --dport 22 -j LOG iptables -A INPUT -p tcp -s officefw --dport 22 -j ACCEPT
Those are the only iptables rules present on the server accepting the connections.
Both of those appear to indicate that the server in the colo facility is receiving the SYN packets. What possible reasons are there that it would not reply with SYN+ACK?
19:08:43.751579 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705398041 0,nop,wscale 7> 19:08:46.751136 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705401041 0,nop,wscale 7> 19:08:52.749305 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705407041 0,nop,wscale 7> 19:09:04.747287 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705419041 0,nop,wscale 7> 19:09:28.741854 IP officefw.57948 > remoteserver.ssh: S 3347102294:3347102294(0) win 5840 <mss1460,sackOK,timestamp 2705443041 0,nop,wscale 7> 19:09:28.742540 IP remoteserver.ssh > officefw.57948: S 3324501337:3324501337(0) ack 3347102295 win 5792 <mss1460,sackOK,timestamp 15960089 2705443041,nop,wscale 7> 19:09:28.783886 IP officefw.57948 > remoteserver.ssh: . ack 1 win 46 <nop,nop,timestamp 2705443083 15960089> 19:09:28.789814 IP remoteserver.ssh > officefw.57948: P 1:21(20) ack 1 win 46 <nop,nop,timestamp 159601372705443083> 19:09:28.829114 IP officefw.57948 > remoteserver.ssh: . ack 21 win 46 <nop,nop,timestamp 2705443129 15960137>
Sep 4 19:08:43 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41807 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 Sep 4 19:08:46 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41808 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 Sep 4 19:08:52 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41809 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 Sep 4 19:09:04 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41810 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 Sep 4 19:09:28 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=60 TOS=0x00 PREC=0x00 TTL=56 ID=41811 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 Sep 4 19:09:28 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=52 TOS=0x00 PREC=0x00 TTL=56 ID=41812 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=46 RES=0x00 ACK URGP=0 Sep 4 19:09:28 remoteserver kernel: IN=eth0 OUT= MAC=00:30:48:97:5a:3a:00:0a:b8:8e:53:29:08:00 SRC=officefw DST=remoteserver LEN=52 TOS=0x00 PREC=0x00 TTL=56 ID=41813 DF PROTO=TCP SPT=57948 DPT=22 WINDOW=46 RES=0x00 ACK URGP=0