[CentOS] tcpdump loses lots of packets

Sun Aug 14 10:20:25 UTC 2016
Anand Buddhdev <anandb at ripe.net>

Hi folks,

I've got a Dell R320 running CentOS 7, and a 10G NIC. I'm running a DNS
server on it, for testing. As part of my testing, I'm attempting to
capture all the DNS queries arriving on the server, using tcpdump.
However, tcpdump's performance is abysmal, and it loses lots of the
packets. Here's the output when I stop the capture:

# tcpdump -nn -i p1p1 -w /dev/shm/queries.pcap -B 65536 dst host
tcpdump: listening on p1p1, link-type EN10MB (Ethernet), capture size
65535 bytes
^C4096558 packets captured
6882162 packets received by filter
2785604 packets dropped by kernel

The number 6882162 is exactly the number of DNS queries I am sending
from another server (the source). The filter is seeing them. However,
not all of them make it into the pcap file. First I was writing to the
disk, and thought it might be a bottleneck, so I tried to write to
/dev/shm. You'll see I've also increased the buffer size with -B, but
none of this helps.

The packet rate is also not that high. From the sending side, this is
what I have:

# tcpreplay -i qtx:p1p1 5min.pcap
Actual: 6882162 packets (620533334 bytes) sent in 317.08 seconds.
Rated: 1951900.0 Bps, 15.61 Mbps, 21648.97 pps
Flows: 4254416 flows, 13382.96 fps, 6882162 flow packets, 0 non-flow

This is a very modest rate.

Any ideas why tcpdump loses so many packets? And how to make it do better?