Hi folks,
I've got a Dell R320 running CentOS 7, and a 10G NIC. I'm running a DNS server on it, for testing. As part of my testing, I'm attempting to capture all the DNS queries arriving on the server, using tcpdump. However, tcpdump's performance is abysmal, and it loses lots of the packets. Here's the output when I stop the capture:
# tcpdump -nn -i p1p1 -w /dev/shm/queries.pcap -B 65536 dst host 193.0.14.129 tcpdump: listening on p1p1, link-type EN10MB (Ethernet), capture size 65535 bytes ^C4096558 packets captured 6882162 packets received by filter 2785604 packets dropped by kernel
The number 6882162 is exactly the number of DNS queries I am sending from another server (the source). The filter is seeing them. However, not all of them make it into the pcap file. First I was writing to the disk, and thought it might be a bottleneck, so I tried to write to /dev/shm. You'll see I've also increased the buffer size with -B, but none of this helps.
The packet rate is also not that high. From the sending side, this is what I have:
# tcpreplay -i qtx:p1p1 5min.pcap Actual: 6882162 packets (620533334 bytes) sent in 317.08 seconds. Rated: 1951900.0 Bps, 15.61 Mbps, 21648.97 pps Flows: 4254416 flows, 13382.96 fps, 6882162 flow packets, 0 non-flow
This is a very modest rate.
Any ideas why tcpdump loses so many packets? And how to make it do better?
Regards, Anand
On 08/14/2016 03:20 AM, Anand Buddhdev wrote:
The number 6882162 is exactly the number of DNS queries I am sending from another server (the source). The filter is seeing them. However, not all of them make it into the pcap file.
Try specifying "ether host <src mac>" and compare the pcap files. How are you counting the number of packets in the pcap file?
Capturing 10 gigabit traffic with no packet loss at line speed is difficult at best. Make sure that you've configured the IRQ affinity properly on the sending and receiving sides to start.
On Aug 14, 2016 11:52 AM, "Gordon Messmer" gordon.messmer@gmail.com wrote:
On 08/14/2016 03:20 AM, Anand Buddhdev wrote:
The number 6882162 is exactly the number of DNS queries I am sending from another server (the source). The filter is seeing them. However, not all of them make it into the pcap file.
Try specifying "ether host <src mac>" and compare the pcap files. How are you counting the number of packets in the pcap file? _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 14/08/16 12:20, Anand Buddhdev wrote:
Hi folks,
I've discovered something. See below:
The packet rate is also not that high. From the sending side, this is what I have:
# tcpreplay -i qtx:p1p1 5min.pcap
If I send packets without qtx, like this:
tcpreplay -i p1p1 5min.pcap
then tcpdump on the receiving box has no problem, and keeps up happily with the queries, and receives all of them into the pcap file. It seems like the qtx module is somehow interfering with the packet capture, but I don't know how or why yet.
Anyway, for low packet rates, such as 20,000 q/s, qtx isn't necessary, so I will not use it when I want to do packet captures.
Regards, Anand
Hello,
I've found it is helpful to limit the length of the packet you are capturing by using something like -s 256.
On 08/14/2016 06:04 PM, Anand Buddhdev wrote:
On 14/08/16 12:20, Anand Buddhdev wrote:
Hi folks,
I've discovered something. See below:
The packet rate is also not that high. From the sending side, this is what I have:
# tcpreplay -i qtx:p1p1 5min.pcap
If I send packets without qtx, like this:
tcpreplay -i p1p1 5min.pcap
then tcpdump on the receiving box has no problem, and keeps up happily with the queries, and receives all of them into the pcap file. It seems like the qtx module is somehow interfering with the packet capture, but I don't know how or why yet.
Anyway, for low packet rates, such as 20,000 q/s, qtx isn't necessary, so I will not use it when I want to do packet captures.
Regards, Anand _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Aug 14, 2016, at 6:20 AM, Anand Buddhdev anandb@ripe.net wrote:
Any ideas why tcpdump loses so many packets?
Saw your nanog posts...
How many RX queues are configured? What does 'ethtool -S p1p1' show? Any discarded packets in the RX queue(s)?
-
Anand Buddhdev -
Gordon Messmer -
Mike Mohr -
Steve Clark -
Steven Tardy