We've started having a problem with a CentOS 7 server. It looses its IPv6 address, if I understand this issue correctly. We can get in, if we do ssh -4, though.
In the logs, I'm seeing this about twice an hour: <warn> (pid 98466) unhandled DHCP event for interface ens3f0
Now, in googling, I get very few hits putting quotes around "unhanded dhcp exception" - in fact, the only one I found that seemed to talk about it was from someone's slackware box, where there was some sort of configuration, perhaps similar to ifcfg-<if>, and they were telling that person to remove it, because it conflicted with what Networkmanager was trying to do, leaving it in a confused state.
Any thoughts?
mark
On 10 December 2015 at 15:00, m.roth@5-cent.us wrote:
We've started having a problem with a CentOS 7 server. It looses its IPv6 address, if I understand this issue correctly. We can get in, if we do ssh -4, though.
In the logs, I'm seeing this about twice an hour: <warn> (pid 98466) unhandled DHCP event for interface ens3f0
Now, in googling, I get very few hits putting quotes around "unhanded dhcp exception" - in fact, the only one I found that seemed to talk about it was from someone's slackware box, where there was some sort of configuration, perhaps similar to ifcfg-<if>, and they were telling that person to remove it, because it conflicted with what Networkmanager was trying to do, leaving it in a confused state.
Any thoughts?
Well there's not much to go on here so you're going to have to provide more detail ...
Does the system lose all ipv6 connectivity (is the fe80:: address uncontactable for instance)
Does ip addr sh still show the expected global IPv6 address and it's just not responding?
Are you using a static ipv6, dhcp ipv6 or SLAAC ipv6 configuration?
Does nmcli c sh <conn-name> still have NM thinking there is an address or does that show it as gone as well?
Start with those in mind as initial questions for diagnostics...
James Hogarth wrote:
On 10 December 2015 at 15:00, m.roth@5-cent.us wrote:
We've started having a problem with a CentOS 7 server. It looses its IPv6 address, if I understand this issue correctly. We can get in, if we do ssh -4, though.
In the logs, I'm seeing this about twice an hour: <warn> (pid 98466) unhandled DHCP event for interface ens3f0
Now, in googling, I get very few hits putting quotes around "unhanded dhcp exception" - in fact, the only one I found that seemed to talk about it was from someone's slackware box, where there was some sort of configuration, perhaps similar to ifcfg-<if>, and they were telling that person to remove it, because it conflicted with what Networkmanager was trying to do, leaving it in a confused state.
Well there's not much to go on here so you're going to have to provide more detail ...
Actually, the problem affected one CentOS 6 and one CentOS 7 server.
Does the system lose all ipv6 connectivity (is the fe80:: address uncontactable for instance)
Yes. Each lost their IPv6 address.
Does ip addr sh still show the expected global IPv6 address and it's just not responding?
Are you using a static ipv6, dhcp ipv6 or SLAAC ipv6 configuration?
DHCP via dibbler.
Does nmcli c sh <conn-name> still have NM thinking there is an address or does that show it as gone as well?
<snip> I passed that along to my manager, and he says it's a *really* useful thing to know, so thanks, muchly.
Oh, and Gordon - switch isn't ours. These boxen are in the datacenter, and another division runs that. Probably big Cisco items.
Btw, one of them, at least, had two drops going into it. Removing one seems to have helped.
mark
On Thu, 10 Dec 2015 16:00, m.roth@... wrote:
We've started having a problem with a CentOS 7 server. It looses its IPv6 address, if I understand this issue correctly. We can get in, if we do ssh -4, though.
In the logs, I'm seeing this about twice an hour: <warn> (pid 98466) unhandled DHCP event for interface ens3f0
Now, in googling, I get very few hits putting quotes around "unhanded dhcp exception" - in fact, the only one I found that seemed to talk about it was from someone's slackware box, where there was some sort of configuration, perhaps similar to ifcfg-<if>, and they were telling that person to remove it, because it conflicted with what Networkmanager was trying to do, leaving it in a confused state.
Any thoughts?
mark
My first thought upon reading this was: Well, let's block / drop the irritating packets via firewall / iptables.
Is the source of these packets allowed to contact your box at all? - No : then block it fully, ipv4 and ipv6 - Yes: block all dhcpv4 / dhcpv6 / radv traffic to and from this source. or even more aggressive: first block this box, second only open the minimum required ports to that box.
IMHO, Networkmanager(and its underlaying helpers) should be much more carefull in handling Router / DHCP stuff. It's biggest niggle for me is a missing white- and black-list for (dis-)allowed routers / dhcp-servers.
Is this the "Right(tm)" thing to do? Dunno, but that would be my gut-telling.
- Yamaban
Yamaban wrote:
On Thu, 10 Dec 2015 16:00, m.roth@... wrote:
We've started having a problem with a CentOS 7 server. It looses its IPv6 address, if I understand this issue correctly. We can get in, if we do ssh -4, though.
In the logs, I'm seeing this about twice an hour: <warn> (pid 98466) unhandled DHCP event for interface ens3f0
Now, in googling, I get very few hits putting quotes around "unhanded dhcp exception" - in fact, the only one I found that seemed to talk about it was from someone's slackware box, where there was some sort of configuration, perhaps similar to ifcfg-<if>, and they were telling that person to remove it, because it conflicted with what Networkmanager was trying to do, leaving it in a confused state.
My first thought upon reading this was: Well, let's block / drop the irritating packets via firewall / iptables.
Is the source of these packets allowed to contact your box at all?
Yes. Btw, this is a box in a datacenter, hardwired. It's getting its IP, both 4 and 6 via dhcp from another of our servers in that rack.
- No : then block it fully, ipv4 and ipv6
- Yes: block all dhcpv4 / dhcpv6 / radv traffic to and from this source. or even more aggressive: first block this box, second only open the minimum required ports to that box.
IMHO, Networkmanager(and its underlaying helpers) should be much more carefull in handling Router / DHCP stuff. It's biggest niggle for me is a missing white- and black-list for (dis-)allowed routers / dhcp-servers.
Is this the "Right(tm)" thing to do? Dunno, but that would be my gut-telling.
The other admin I work with was working on it, and it seems to be good. However, since he restarted the network, I do find this in the logs: Dec 10 16:06:01 <server> dhclient[96610]: PRC: Renewing lease on ens3f0. Dec 10 16:06:01 <server> dhclient[96610]: XMT: Renew on ens3f0, interval 10900ms. Dec 10 16:06:01 <server> dhclient[96610]: RCV: Reply message on ens3f0 from fe80::: 222:64ff:fef6:d30c. Dec 10 16:06:01 <server> dhclient[96610]: IA_NA status code NoBinding: "Who are you? Do I know you?" Dec 10 16:06:01 <server> dhclient[96610]: XMT: Request on ens3f0, interval 910ms. Dec 10 16:06:01 <server> dhclient[96610]: RCV: Reply message on ens3f0 from fe80:::<addr> Dec 10 16:06:01 <server> dhclient[96610]: IA_NA status code Success: "Assigned fixx ed address." Dec 10 16:06:01 <server> NetworkManager[96668]: <warn> (pid 96610) unhandled DHCP event for interface ens3f0
Oh, yes, not sure who the 222: address is, but the one I've replaced with <addr> is not merely a valid one, it's the server running dibbler.
So, I'm not sure what the "unhandled DHCP event is.
mark
On 12/10/2015 01:19 PM, m.roth@5-cent.us wrote:
However, since he restarted the network, I do find this in the logs: Dec 10 16:06:01 <server> dhclient[96610]: PRC: Renewing lease on ens3f0. ... Dec 10 16:06:01 <server> dhclient[96610]: RCV: Reply message on ens3f0 from fe80::: 222:64ff:fef6:d30c. Dec 10 16:06:01 <server> dhclient[96610]: IA_NA status code NoBinding: "Who are you? Do I know you?" ... Oh, yes, not sure who the 222: address is, but the one I've replaced with <addr> is not merely a valid one, it's the server running dibbler.
So, I'm not sure what the "unhandled DHCP event is.
I think what you're seeing is the result of having multiple DHCP servers on the same network. The one at 222 appears to hand out addresses to specific clients, and responds to your requests with the message "..Do I know you?". The client reports that to NetworkManager without a "reason", and NM logs an unhandled DHCP event because it doesn't know what to do with the information that dhclient provided.
On 12/10/2015 07:00 AM, m.roth@5-cent.us wrote:
In the logs, I'm seeing this about twice an hour: <warn> (pid 98466) unhandled DHCP event for interface ens3f0
Might be related to these? The fix for NM bug 739482 is included in NetworkManager-1.0.0-16.git20150121.b4ea599c.el7_1.x86_64, the current version on CentOS 7, but it might not be fully fixed? https://bugzilla.redhat.com/show_bug.cgi?id=1181477 https://bugzilla.gnome.org/show_bug.cgi?id=739482
It looks like that gets logged when the child "dhclient" process sends an event without a reason. If that's the case, then switching from NetworkManager to the old "network" service probably won't help. The problem is that "dhclient" is losing the address, and you'll be using that under the "network" service as well.
-
Gordon Messmer -
James Hogarth -
m.roth＠5-cent.us -
Yamaban