Hi All,
Was troubleshooting some odd VM network issues and discovered that we're seeing dropped packets + retransmissions across multiple domU OS's and dom0 hardware platforms.
xendev01 ~ # tshark -R "tcp.analysis.retransmission " -i vif7.0
Running as user "root" and group "root". This could be dangerous.
Capturing on vif7.0
3.054257 xxx.xxx.xxx.196 -> xxx.xxx.xxx.145 SSH 110 [TCP Fast Retransmission] Encrypted response packet len=44
3.061949 xxx.xxx.xxx.196 -> xxx.xxx.xxx.145 SSH 1434 [TCP Fast Retransmission] Encrypted response packet len=1368
3.383880 xxx.xxx.xxx.196 -> xxx.xxx.xxx.145 SSH 1434 [TCP Fast Retransmission] Encrypted response packet len=1368
3.630911 xxx.xxx.xxx.196 -> xxx.xxx.xxx.145 SSH 1434 [TCP Fast Retransmission] Encrypted response packet len=1368
3.635964 xxx.xxx.xxx.196 -> xxx.xxx.xxx.145 SSH 1434 [TCP Fast Retransmission] Encrypted response packet len=1368
I've confirmed this is happening with linux, windows and pfsense (bsd) domU's. I've turned off every feature I can with ethtool on both the underlying bridge on the host, the vif's, and the eth's inside the domU's. I also see it on traffic inbetween vms on the same host.
The domU sees packet errors on incoming traffic and outgoing looks fine, dumping on the dom0 indicates incoming packets are fine, but the reply from the domU is broken. This does not happen running the exact same VMs on some older xen 4.1.3 hosts. Reproduction is easy (for me at least), any burst of traffic will do it. I've just been running "ps auxf" over ssh to a vm to trigger.
Since I'm seeing it on the host when I sniff the vif, this feels like a bug?
- Nathan