Hello!
I am having a strange issue with CentOS 5.4 that I cannot seem to solve.
Every DNS lookup results in AAAA records being requested first before A records. As a result, this causes a large amount of unnecessary DNS traffic on the network. IPv6 has been completely disabled on these servers:
/etc/modprobe.conf, ipv6 off and net-pf-10 off /etc/sysconfig/network, NETWORKING_IPV6=no
lsmod | grep ipv6 shows the kernel module no longer loaded.
Yet watching TCP dump shows that AAAA records are requested before A records every time a login is requested from one of our local machines to another. Is there some sort of configuration directive I can use to force IPv4 lookups first before IPv6? Or even better, stop IPv6 lookups all together?
On Mon, 2011-04-04 at 09:51 -0500, Russell Jones wrote:
Hello! I am having a strange issue with CentOS 5.4 that I cannot seem to solve. Every DNS lookup results in AAAA records being requested first before A records. As a result, this causes a large amount of unnecessary DNS traffic on the network. IPv6 has been completely disabled on these servers:
Doubtful, if you are seeing AAAA lookups. Does "ip addr" show any IPv6 interfaces?
/etc/modprobe.conf, ipv6 off and net-pf-10 off /etc/sysconfig/network, NETWORKING_IPV6=no lsmod | grep ipv6 shows the kernel module no longer loaded. Yet watching TCP dump shows that AAAA records are requested before A records every time a login is requested from one of our local machines to another
You *only* sees these for login? Perhaps some authentication module you are using is causing them to happen?
Is there some sort of configuration directive I can use to force IPv4 lookups first before IPv6? Or even better, stop IPv6 lookups all together?
I don't believe you see IPv6 lookups from the normal resolver libraries unless there is at least one active IPv6 interface.
Thanks for your help!
Doubtful, if you are seeing AAAA lookups. Does "ip addr" show any
IPv6 interfaces?
No active ipv6 interfaces:
[root@hostname1 ~]# ip addr 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:e0:81:76:ca:52 brd ff:ff:ff:ff:ff:ff inet 172.29.87.20/24 brd 172.29.87.255 scope global eth0 3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000 link/ether 00:e0:81:76:ca:53 brd ff:ff:ff:ff:ff:ff
[root@hostname1 ~]# lsmod | grep -i ipv6 [root@hostname1 ~]#
You *only* sees these for login? Perhaps some authentication module
you are using is causing them to happen?
No, I also see this when doing traceroutes. It's just the "at login" part that it's most prevalent due to the DNS-induced lag time. The server's user authentication is NIS, and I am unaware of any IPv6 NIS configuration that would cause this. But if you are please do let me know, I'm tearing my hair out here and I don't have much left. Example from a traceroute from hostname1 to hostname2. This was nothing more than a "traceroute hostname2" on this same box I just showed as not having any IPv6 interfaces, nor the IPv6 kernel module loaded:
[root@hostname1 ~]# tcpdump -vvvvv 'port 53' tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:07:24.989304 IP (tos 0x0, ttl 64, id 65039, offset 0, flags [DF], proto: UDP (17), length: 60) hostname1.59725 > vdns1-hc.example.com.domain: [bad udp cksum 2bd2!] 26130+ AAAA? hostname2.example.com. (32) 11:07:24.989612 IP (tos 0x0, ttl 62, id 61322, offset 0, flags [DF], proto: UDP (17), length: 111) vdns1-hc.example.com.domain > hostname1.59725: 26130* q: AAAA? hostname2.example.com. 0/1/0 ns: example.com. SOA[|domain] 11:07:24.989666 IP (tos 0x0, ttl 64, id 65040, offset 0, flags [DF], proto: UDP (17), length: 69) hostname1.47865 > vdns1-hc.example.com.domain: [bad udp cksum 9d73!] 33222+ AAAA? hostname2.ioexample.ioroot.tld. (41) 11:07:24.989763 IP (tos 0x0, ttl 64, id 65040, offset 0, flags [DF], proto: UDP (17), length: 72) hostname1.58021 > vdns1-hc.example.com.domain: [bad udp cksum f44f!] 24663+ PTR? 20.251.31.172.in-addr.arpa. (44) 11:07:24.990137 IP (tos 0x0, ttl 62, id 61323, offset 0, flags [DF], proto: UDP (17), length: 140) vdns1-hc.example.com.domain > hostname1.58021: 24663* q: PTR? 20.251.31.172.in-addr.arpa. 1/1/1 20.251.31.172.in-addr.arpa.[|domain] 11:07:24.991182 IP (tos 0x0, ttl 62, id 61324, offset 0, flags [DF], proto: UDP (17), length: 133) vdns1-hc.example.com.domain > hostname1.47865: 33222 NXDomain q: AAAA? hostname2.ioexample.ioroot.tld. 0/1/0 ns: ioexample.ioroot.tld. SOA[|domain] 11:07:24.991214 IP (tos 0x0, ttl 64, id 65041, offset 0, flags [DF], proto: UDP (17), length: 60) hostname1.42778 > vdns1-hc.example.com.domain: [bad udp cksum 5f4a!] 12333+ A? hostname2.example.com. (32) 11:07:24.991665 IP (tos 0x0, ttl 62, id 61325, offset 0, flags [DF], proto: UDP (17), length: 291) vdns1-hc.example.com.domain > hostname1.42778: 12333* q: A? hostname2.example.com. 1/6/5 hostname2.example.com. A hostname2.example.com ns: example.com.[|domain] 11:07:24.991797 IP (tos 0x0, ttl 64, id 65042, offset 0, flags [DF], proto: UDP (17), length: 71) hostname1.38670 > vdns1-hc.example.com.domain: [bad udp cksum 1a59!] 25903+ PTR? 43.43.29.172.in-addr.arpa. (43) 11:07:24.992089 IP (tos 0x0, ttl 62, id 61326, offset 0, flags [DF], proto: UDP (17), length: 137) vdns1-hc.example.com.domain > hostname1.38670: 25903* q: PTR? 43.43.29.172.in-addr.arpa. 1/1/1 43.43.29.172.in-addr.arpa.[|domain] 11:07:24.993030 IP (tos 0x0, ttl 64, id 65043, offset 0, flags [DF], proto: UDP (17), length: 72) hostname1.51119 > vdns1-hc.example.com.domain: [bad udp cksum 161e!] 53743+ PTR? 254.87.29.172.in-addr.arpa. (44) 11:07:24.993608 IP (tos 0x0, ttl 62, id 24219, offset 0, flags [DF], proto: UDP (17), length: 147) vdns1-hc.example.com.domain > hostname1.51119: 53743* q: PTR? 254.87.29.172.in-addr.arpa. 1/1/1 254.87.29.172.in-addr.arpa.[|domain] 11:07:24.993922 IP (tos 0x0, ttl 64, id 65044, offset 0, flags [DF], proto: UDP (17), length: 71) hostname1.49775 > vdns1-hc.example.com.domain: [bad udp cksum 6bab!] 59260+ PTR? 43.43.29.172.in-addr.arpa. (43) 11:07:24.994401 IP (tos 0x0, ttl 62, id 24220, offset 0, flags [DF], proto: UDP (17), length: 137) vdns1-hc.example.com.domain > hostname1.49775: 59260* q: PTR? 43.43.29.172.in-addr.arpa. 1/1/1 43.43.29.172.in-addr.arpa.[|domain]
Notice how before it even attempts an IPv4 lookup it cycles through IPv6 first, appending the domains that are in the "search" path of this boxes' /etc/resolv.conf to the host I did a traceroute to. Then after it exhausts all of its IPv6 lookup attempts, it does IPv4 and of course the first lookup succeeds. With boxes that have 3 or 4 domains on the search path, this causes a large amount unneeded IPv6 DNS traffic.
This behavior doesn't make a whole lot of sense, so I am hoping I'm missing something somewhere to force it to always do IPv4 DNS lookups first, or disable IPv6 lookups completely. All boxes' /etc/resolv.conf aren't special, just name server lines and then the "search" option with internal domains on it. We don't use IPv6 so don't need the functionality.
On similar CentOS 4 boxes with the same configuration on the same network, it does not attempt IPv6 lookups at all with traceroutes or SSH logins.
On 4/4/2011 10:50 AM, Adam Tauno Williams wrote:
On Mon, 2011-04-04 at 09:51 -0500, Russell Jones wrote:
Hello! I am having a strange issue with CentOS 5.4 that I cannot seem to solve. Every DNS lookup results in AAAA records being requested first before A records. As a result, this causes a large amount of unnecessary DNS traffic on the network. IPv6 has been completely disabled on these servers:
Doubtful, if you are seeing AAAA lookups. Does "ip addr" show any IPv6 interfaces?
/etc/modprobe.conf, ipv6 off and net-pf-10 off /etc/sysconfig/network, NETWORKING_IPV6=no lsmod | grep ipv6 shows the kernel module no longer loaded. Yet watching TCP dump shows that AAAA records are requested before A records every time a login is requested from one of our local machines to another
You *only* sees these for login? Perhaps some authentication module you are using is causing them to happen?
Is there some sort of configuration directive I can use to force IPv4 lookups first before IPv6? Or even better, stop IPv6 lookups all together?
I don't believe you see IPv6 lookups from the normal resolver libraries unless there is at least one active IPv6 interface.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Mon, Apr 04, 2011 at 11:34:25AM -0500, Russell Jones wrote:
[root@hostname1 ~]# tcpdump -vvvvv 'port 53' tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:07:24.989304 IP (tos 0x0, ttl 64, id 65039, offset 0, flags [DF], proto: UDP (17), length: 60) hostname1.59725 > vdns1-hc.example.com.domain: [bad udp cksum 2bd2!] 26130+ AAAA? hostname2.example.com. (32)
Check https://bugzilla.redhat.com/show_bug.cgi?id=140528 and see if that resolves your issue.
Thank you, but unfortunately this is a different issue. These boxes do not run bind, they resolve their DNS queries via dedicated bind servers on the network. Configuring the bind servers on the network a different way still would not stop the IPv6 traffic I am showing in the TCP dump from being sent.
On 4/4/2011 11:54 AM, Stephen Harris wrote:
On Mon, Apr 04, 2011 at 11:34:25AM -0500, Russell Jones wrote:
[root@hostname1 ~]# tcpdump -vvvvv 'port 53' tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:07:24.989304 IP (tos 0x0, ttl 64, id 65039, offset 0, flags [DF], proto: UDP (17), length: 60) hostname1.59725> vdns1-hc.example.com.domain: [bad udp cksum 2bd2!] 26130+ AAAA? hostname2.example.com. (32)
Check https://bugzilla.redhat.com/show_bug.cgi?id=140528 and see if that resolves your issue.
centos-bounces@centos.org wrote:
Thank you, but unfortunately this is a different issue. These boxes do not run bind, they resolve their DNS queries via dedicated bind servers on the network. Configuring the bind servers on the network a different way still would not stop the IPv6 traffic I am showing in the TCP dump from being sent.
Things described in the bug report are
Adding NETWORKING_IPV6=yes to /etc/sysconfig/network prevents the queries to root nameservers with IPv6 addresses timing out.
And
alias net-pf-10 off to /etc/modprobe.conf
Have you tried both of them?
On 4/4/2011 11:54 AM, Stephen Harris wrote:
On Mon, Apr 04, 2011 at 11:34:25AM -0500, Russell Jones wrote:
[root@hostname1 ~]# tcpdump -vvvvv 'port 53' tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:07:24.989304 IP (tos 0x0, ttl 64, id 65039, offset 0, flags [DF], proto: UDP (17), length: 60) hostname1.59725> vdns1-hc.example.com.domain: [bad udp cksum 2bd2!] 26130+ AAAA? hostname2.example.com. (32)
Check https://bugzilla.redhat.com/show_bug.cgi?id=140528 and see if that resolves your issue.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Insert spiffy .sig here: Life is complex: it has both real and imaginary parts.
//me ******************************************************************* This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept for the presence of computer viruses. www.Hubbell.com - Hubbell Incorporated**
Thanks.
Yes, the modules are disabled via /etc/modprobe.conf: alias net-pf-10 off alias ipv6 off
The issue at stake here is not queries timing out, as these aren't even external network queries, it's the queries being sent to begin with. We have thousands of CentOS 5 boxes all doing 3 or more IPv6 DNS queries for 1 IPv4 host. These aren't wanted or needed on the network, and it's causing a large amount of unneeded traffic and strain on our DNS servers. We need the traffic to go away, we don't want any IPv6 DNS queries at all, as they are useless to us. They should not be sent when IPv6 is disabled in both networking and kernel modules, and no IPv6 address exists on the interfaces, yet they still are.
I'm unsure how enabling IPv6 via /etc/sysconfig/network is going to make the IPv6 DNS queries stop, but I tried it anyway:
[root@hostname1 sysconfig]# nano -w network [root@hostname1 sysconfig]# service network restart Shutting down interface eth0: [ OK ] Shutting down loopback interface: [ OK ] FATAL: Module off not found. CRITICAL : [ipv6_test] Kernel is not compiled with IPv6 support Bringing up loopback interface: [ OK ] Bringing up interface eth0: [ OK ] FATAL: Module off not found. CRITICAL : [ipv6_test] Kernel is not compiled with IPv6 support
Re-enabling the IPv6 kernel modules will just put us back to where we were to begin with. Any other ideas on how to make the AAAA queries stop?
On 4/4/2011 12:10 PM, Brunner, Brian T. wrote:
centos-bounces@centos.org wrote:
Thank you, but unfortunately this is a different issue. These boxes do not run bind, they resolve their DNS queries via dedicated bind servers on the network. Configuring the bind servers on the network a different way still would not stop the IPv6 traffic I am showing in the TCP dump from being sent.
Things described in the bug report are
Adding NETWORKING_IPV6=yes to /etc/sysconfig/network prevents the queries to root nameservers with IPv6 addresses timing out.
And
alias net-pf-10 off to /etc/modprobe.conf
Have you tried both of them?
On 4/4/2011 11:54 AM, Stephen Harris wrote:
On Mon, Apr 04, 2011 at 11:34:25AM -0500, Russell Jones wrote:
[root@hostname1 ~]# tcpdump -vvvvv 'port 53' tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
11:07:24.989304 IP (tos 0x0, ttl 64, id 65039, offset 0, flags [DF], proto: UDP (17), length: 60) hostname1.59725> vdns1-hc.example.com.domain: [bad udp cksum 2bd2!] 26130+ AAAA? hostname2.example.com. (32)
Check https://bugzilla.redhat.com/show_bug.cgi?id=140528 and see if that resolves your issue.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Insert spiffy .sig here: Life is complex: it has both real and imaginary parts.
//me
This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept for the presence of computer viruses. www.Hubbell.com - Hubbell Incorporated**
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Russell Jones wrote:
Thanks.
Yes, the modules are disabled via /etc/modprobe.conf: alias net-pf-10 off alias ipv6 off
The issue at stake here is not queries timing out, as these aren't even external network queries, it's the queries being sent to begin with. We
<snip> Really dumb question, which may have been answered: what's in /etc/nsswitch?
mark
No such thing as dumb questions when you're asking other people for help :-)
Here's the whole file:
================= passwd: files nis shadow: files nis group: files nis
hosts: files dns nis
bootparams: nisplus [NOTFOUND=return] files
ethers: files netmasks: files networks: files protocols: files nis rpc: files services: files nis
netgroup: files nis
publickey: nisplus
#automount: nis automount: ldap aliases: files nisplus =======================
On 4/4/2011 1:22 PM, m.roth@5-cent.us wrote:
Russell Jones wrote:
Thanks.
Yes, the modules are disabled via /etc/modprobe.conf: alias net-pf-10 off alias ipv6 off
The issue at stake here is not queries timing out, as these aren't even external network queries, it's the queries being sent to begin with. We
<snip> Really dumb question, which may have been answered: what's in /etc/nsswitch?
mark
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Mon, Apr 4, 2011 at 10:51 AM, Russell Jones rjones@eggycrew.com wrote:
I am having a strange issue with CentOS 5.4 that I cannot seem to solve.
Every DNS lookup results in AAAA records being requested first before A records. As a result, this causes a large amount of unnecessary DNS traffic on the network. IPv6 has been completely disabled on these servers:
/etc/modprobe.conf, ipv6 off and net-pf-10 off /etc/sysconfig/network, NETWORKING_IPV6=no
lsmod | grep ipv6 shows the kernel module no longer loaded.
Yet watching TCP dump shows that AAAA records are requested before A records every time a login is requested from one of our local machines to another. Is there some sort of configuration directive I can use to force IPv4 lookups first before IPv6? Or even better, stop IPv6 lookups all together?
Disabling ipv6 transport cannot prevent applications from making ipv6 queries - short of recompiling them as ipv4-only applications or having applications check whether there is a non-link-local ipv6 address before making an ipv6 query. I've seen these checks discussed but I don't think that they've been implemented - or, if they've been implemented, backported to CentOS 5. It's been going on for a while:
https://www.redhat.com/archives/redhat-list/2009-March/msg00067.html
Thank you!
If forcing it to stop system-wide is not possible, is there any way of forcing IPv4 lookups to occur first then?
On 4/4/2011 5:34 PM, Tom H wrote:
On Mon, Apr 4, 2011 at 10:51 AM, Russell Jonesrjones@eggycrew.com wrote:
I am having a strange issue with CentOS 5.4 that I cannot seem to solve.
Every DNS lookup results in AAAA records being requested first before A records. As a result, this causes a large amount of unnecessary DNS traffic on the network. IPv6 has been completely disabled on these servers:
/etc/modprobe.conf, ipv6 off and net-pf-10 off /etc/sysconfig/network, NETWORKING_IPV6=no
lsmod | grep ipv6 shows the kernel module no longer loaded.
Yet watching TCP dump shows that AAAA records are requested before A records every time a login is requested from one of our local machines to another. Is there some sort of configuration directive I can use to force IPv4 lookups first before IPv6? Or even better, stop IPv6 lookups all together?
Disabling ipv6 transport cannot prevent applications from making ipv6 queries - short of recompiling them as ipv4-only applications or having applications check whether there is a non-link-local ipv6 address before making an ipv6 query. I've seen these checks discussed but I don't think that they've been implemented - or, if they've been implemented, backported to CentOS 5. It's been going on for a while:
https://www.redhat.com/archives/redhat-list/2009-March/msg00067.html _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Tue, Apr 5, 2011 at 6:52 PM, Russell Jones rjones@eggycrew.com wrote:
Thank you!
If forcing it to stop system-wide is not possible, is there any way of forcing IPv4 lookups to occur first then?
You're welcome.
In the case of traceroute, there shouldn't be any AAAA DNS requests when specifying ipv4 transport ("-4").
Perhaps you other applications have a similar option...
On Tue, Apr 05, 2011 at 07:46:32PM -0400, Tom H wrote:
In the case of traceroute, there shouldn't be any AAAA DNS requests when specifying ipv4 transport ("-4").
Umm, no. The transport protocol is irrelevant to the query. You can make AAAA queries over IPv4. Indeed I do that all the time.
On Tue, Apr 5, 2011 at 7:50 PM, Stephen Harris lists@spuddy.org wrote:
On Tue, Apr 05, 2011 at 07:46:32PM -0400, Tom H wrote:
In the case of traceroute, there shouldn't be any AAAA DNS requests when specifying ipv4 transport ("-4").
Umm, no. The transport protocol is irrelevant to the query. You can make AAAA queries over IPv4. Indeed I do that all the time.
You can make ipv6 queries on ipv4 (which is what's happening to the OP since he's disabled ipv6 on his box) but I've just checked and traceroute doesn't make an AAAA query (unless I was in too big a hurry and missed it!).
Whether other applications have an equivalent option and are this "intelligent" will have to be checked app by app, although it would be the logical behavior.
-
Adam Tauno Williams -
Brunner, Brian T. -
m.roth＠5-cent.us -
Russell Jones -
Stephen Harris -
Tom H