I am having trouble with recovery. Today due to electrical work I powered down my networks Router / DHCP server.
My Centos 7 host machines lost their DHCP lease (they are actually static leases). Once I power my Router / DHCP server back up none of my virtual machine were accessible. It appears that when the DHCP lease was lost on the virt server the bridge of the KVM guest were all losts. A 'br show' had none of the guests anymore. So each guest was sending out DHCP requests as expected but since the guest was no longer bridged the packets went no-where.
How do I handle proper recovery from this?
The sequence on the host seemed to be:
07:00:00 I powered off the DHCP server and router for electrical work
Jun 26 10:08:02 vm-server dhclient[973]: DHCPREQUEST on br0 to 192.168.2.1 port 67 (xid=0x7627186d)
requests continue for quite some hours and then NetworkManager chimes in.
Jun 26 14:55:23 vm-server NetworkManager[840]: <info> [1498503323.1518] dhcp4 (br0): state changed bound -> expire
Jun 26 14:55:23 vm-server NetworkManager[840]: <info> [1498503323.1681] dhcp4 (br0): canceled DHCP transaction, DHCP client pid 973 Jun 26 14:55:23 vm-server NetworkManager[840]: <info> [1498503323.1681] dhcp4 (br0): state changed expire -> done Jun 26 14:55:23 vm-server NetworkManager[840]: <info> [1498503323.1684] device (br0): scheduling DHCPv4 restart in 120 seconds, 3 tries left (reason: lease expired)
(looks like 3 tries of something)
Jun 26 15:00:52 vm-server NetworkManager[840]: <warn> [1498503652.9531] dhcp4 (br0): request timed out Jun 26 15:00:52 vm-server NetworkManager[840]: <info> [1498503652.9532] dhcp4 (br0): state changed unknown -> timeout Jun 26 15:00:52 vm-server NetworkManager[840]: <info> [1498503652.9694] dhcp4 (br0): canceled DHCP transaction, DHCP client pid 15937 Jun 26 15:00:52 vm-server NetworkManager[840]: <info> [1498503652.9694] dhcp4 (br0): state changed timeout -> done Jun 26 15:00:52 vm-server NetworkManager[840]: <info> [1498503652.9697] device (br0): scheduling DHCPv4 restart in 120 seconds, 1 tries left (reason: lease expired) Jun 26 15:02:52 vm-server NetworkManager[840]: <info> [1498503772.9548] dhcp4 (br0): activation: beginning transaction (timeout in 45 seconds) Jun 26 15:02:52 vm-server NetworkManager[840]: <info> [1498503772.9566] dhcp4 (br0): dhclient started with pid 15994 Jun 26 15:02:52 vm-server dhclient[15994]: DHCPDISCOVER on br0 to 255.255.255.255 port 67 interval 4 (xid=0x54083ff1) Jun 26 15:03:37 vm-server NetworkManager[840]: <warn> [1498503817.9818] dhcp4 (br0): request timed out Jun 26 15:03:37 vm-server NetworkManager[840]: <info> [1498503817.9819] dhcp4 (br0): state changed unknown -> timeout Jun 26 15:03:37 vm-server NetworkManager[840]: <info> [1498503817.9981] dhcp4 (br0): canceled DHCP transaction, DHCP client pid 15994 Jun 26 15:03:37 vm-server NetworkManager[840]: <info> [1498503817.9981] dhcp4 (br0): state changed timeout -> done Jun 26 15:03:37 vm-server NetworkManager[840]: <info> [1498503817.9984] device (br0): state change: activated -> failed (reason 'ip-config-unavailable') [100 120 5] Jun 26 15:03:37 vm-server NetworkManager[840]: <info> [1498503817.9987] manager: NetworkManager state is now CONNECTED_LOCAL Jun 26 15:03:38 vm-server NetworkManager[840]: <warn> [1498503818.0457] device (br0): Activation: failed for connection 'br0' Jun 26 15:03:38 vm-server kernel: device enp3s0 left promiscuous mode Jun 26 15:03:38 vm-server kernel: br0: port 1(enp3s0) entered disabled state Jun 26 15:03:38 vm-server NetworkManager[840]: <info> [1498503818.0473] device (br0): detached bridge port enp3s0 Jun 26 15:03:38 vm-server kernel: device vnet0 left promiscuous mode Jun 26 15:03:38 vm-server kernel: br0: port 2(vnet0) entered disabled state Jun 26 15:03:38 vm-server NetworkManager[840]: <info> [1498503818.0483] device (br0): detached bridge port vnet0 Jun 26 15:03:38 vm-server kernel: device vnet2 left promiscuous mode Jun 26 15:03:38 vm-server kernel: br0: port 3(vnet2) entered disabled state Jun 26 15:03:38 vm-server NetworkManager[840]: <info> [1498503818.0493] device (br0): detached bridge port vnet2 Jun 26 15:03:38 vm-server kernel: device vnet3 left promiscuous mode
When I powered back on at:
Jun 26 16:45:29 vm-server dhclient[26539]: DHCPOFFER from 192.168.2.1 Jun 26 16:45:29 vm-server dhclient[26539]: DHCPACK from 192.168.2.1 (xid=0x4c7b48be)
None of the bridges that NetworkManager decided to detach were restored.
Is this the case of being bitten by NetworkManager once again? Not sure why I am running it other than it is the default and I don't remember if it is responsible for setting up the bridges.