[CentOS-virt] Seeing dropped packets / tcp retrans on latest 4.4.1-10el6

[George Dunlap]

[cc'ing xen-users to see if anyone there has any familiarity with ebtables]

On Fri, Apr 17, 2015 at 8:20 PM, Nathan March <nathan at gt.net> wrote:
> Hi All,
>
> I've tracked this down... We do rate limiting of our vms with a mix of ebtables/tc.
>
> Running these commands (replace vif1.0 with the correct vif for your VM) will reproduce this:
>
> ebtables -A FORWARD -i vif1.0 -j mark --set-mark 990 --mark-target CONTINUE
>
> tc qdisc add dev bond0 root handle 1: htb default 2
> tc class add dev bond0 parent 1: classid 1:0 htb rate 10000mbit
>
> tc class add dev bond0 parent 1: classid 1:990 htb rate 10000mbit
> tc filter add dev bond0 protocol ip parent 1:0 prio 990 handle 990 fw flowid 1:990
>
> Note that the speed limits being applied here are 10gb and I'm testing this on a 1gb network, so TC shouldn't really be doing anything here except letting the packets through. These same commands worked fine on gentoo xen 4.1 / kernel 3.2.57, compared to this now not working on centos xen 4.4.1 / kernel 3.10.68.

So just to be clear, we have 3 variables to consider here?

* gentoo -> CentOS
* Xen 4.1 -> Xen 4.4
* kernel 3.2.57 -> 3.10.68

Unfortunately I'm not very familiar with ebtables, so I don't have a
clear idea what sort of thing might cause duplicate ACKs.

Are you able to narrow down any of those?

You can find CentOS packages for Xen 4.2 and kernel 3.4 here:

http://vault.centos.org/6.4/xen4/x86_64/Packages/

If you could build a more recent kernel and see if it's been fixed,
that might be helpful as well.

 -George

>
> Easiest way to reproduce is simply generate a large file, scp it to a remote host and on the remote host run:
> tshark -Y "tcp.analysis.duplicate_ack_num"
>
> If you run the ssh in a loop + tshark in another window, you can see the Dup ACK's begin immediately after adding the last filter rule:
>
> 25790294 1752.756733 xxx.xxx.xxx.13 -> xxx.xxx.xxx.205 TCP 78 [TCP Dup ACK 25790286#4] ssh > 51515 [ACK] Seq=15994 Ack=50769840 Win=1544704 Len=0 TSval=738150929 TSecr=4294944346 SLE=50785768 SRE=50790596
> 25790296 1752.756742 xxx.xxx.xxx.13 -> xxx.xxx.xxx.205 TCP 78 [TCP Dup ACK 25790286#5] ssh > 51515 [ACK] Seq=15994 Ack=50769840 Win=1544704 Len=0 TSval=738150929 TSecr=4294944346 SLE=50785768 SRE=50792044
>
> - Nathan
>
> _______________________________________________
> CentOS-virt mailing list
> CentOS-virt at centos.org
> http://lists.centos.org/mailman/listinfo/centos-virt