[CentOS] CentOS 7, systemd, NetworkMangler, oh, my

[James Hogarth]

On 13 February 2017 at 15:35,  <m.roth at 5-cent.us> wrote:
> My manager tells me a system in the datacenter is down. I go down there,
> and plug in a monitor-on-a-stick and keyboard. It's up, but no network. I
> try systemctl restart NetworkManager several times, and ip a shows *no*
> change.
>
> Finally, I do an ifdown, followed by an ifup, and everything's wonderful.
>
> My manager thinks that the NM daemon thinks everything's fine, and
> there've been no changes, so it does nothing. He suggests that it might
> have to be stopped, then started, rather than restarted.
>
> This is completely unacceptable behavior, since it leave the system with
> no network connection. Pre-systemd, as we all know, restart *RESTARTED*
> the damn thing.
>
> Is there some Magic (#insert "pixie-dust-sparkles") incantation, either
> restarting NetworkManager, or using nm-cli, to force it to perform the
> expected actions?
>


I'd be interested in the journal from the NetworkManager restart as
that's not the way it behaves ... it uses the netlink API to get state
and not it's own internal tracker of state (ie doing an ip link down
will reflect in nmcli output) ... a restart of NetworkManager should
not ignore interfaces but rather bring the system to the on disk
configured state ... and a quick check it doesn't override ExecRestart
in the unit file to do a reload or similar instead ...

And indeed a quick test in a VM shows nmcli device status correctly
changing between connected and unavailable when doing ip link set eth0
down/up

Do note that on a NM based system ifup and ifdown are effectively
aliases to nmcli conn down and nmcli conn up

nmcli conn down "connection name" will make it disconnected
nmcli conn up "connection mame" will bring it back to connected

there is a slight interesting difference between using nmcli and ip
link set though ...

with ip link set down <interface> the interface is marked
administratively down (as if you've pulled the cable) but nmcli conn
down "connection name" will unconfigure the interface but leave it in
an UP state ... just without an IP address etc

anyway that's just an interesting diversion on behavioural differences

NM won't change an interface state without some sort of event though
(manual or virtual cable pulled etc), and if you have a case where it
*has* done that then you have found a bug that would be great to get
reported

TL;DR: cannot reproduce, need logs to determine what happened without
a working crystal ball