Hi,
Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a reboot of the first server the network was unavailable. In IPMI console everything except the network was looking good. Network was unreachable. No errors in NetworkManager. I also restarted NetworkManager, but it did not help. Then I discovered that the default gateway suddenly was missing.
Then I rebooted the server one more time, but network was still down.
Then both myself and a technician in my datcenter was debugging this (I had to wake him up in the middle of the night, costing me a lot of money), without finding any reason for why the default gateway was missing after reboot.
Then we rebooted the server a third time, and all of a sudden the problem was gone and the default gateway was back.
Then, after this, I also upgraded my second CentOS 8 server to CentOS 8.1 and did a reboot. And the very same thing happened to this server after reboot! The default gateway was missing after reboot, and network was down. Then I did a extra reboot of this server also, and when it came back up everything was working correct and the default gateway was back.
So the first server I needed to reboot two extra times to have the default gateway back and network working. And the second server I needed to reboot only one extra time for the problem to be solved.
The two boxes is not VPSs or anything, but bare metal dedicated servers.
Also prior to upgrading to CentOS 8.1, these two boxes has been rebooted serveral times previously, without any problems at all. Only after upgrading to CentOS 8.1 this happened on both of them for the first time.
I can't believe I am the only one that experience this? My guess is that this is a unknown random intermittent bug in CentOS 8.1 that kicked in. I just hope this does not continue to happen in the future. If you experienced the same, please share it with a reply. Thank you!
Kind regards, Asle Ommundsen
On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen aommundsen@gmail.com wrote:
Hi,
Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a reboot of the first server the network was unavailable. In IPMI console everything except the network was looking good. Network was unreachable. No errors in NetworkManager. I also restarted NetworkManager, but it did not help. Then I discovered that the default gateway suddenly was missing.
Then I rebooted the server one more time, but network was still down.
Then both myself and a technician in my datcenter was debugging this (I had to wake him up in the middle of the night, costing me a lot of money), without finding any reason for why the default gateway was missing after reboot.
Then we rebooted the server a third time, and all of a sudden the problem was gone and the default gateway was back.
Then, after this, I also upgraded my second CentOS 8 server to CentOS 8.1 and did a reboot. And the very same thing happened to this server after reboot! The default gateway was missing after reboot, and network was down. Then I did a extra reboot of this server also, and when it came back up everything was working correct and the default gateway was back.
So the first server I needed to reboot two extra times to have the default gateway back and network working. And the second server I needed to reboot only one extra time for the problem to be solved.
The two boxes is not VPSs or anything, but bare metal dedicated servers.
Also prior to upgrading to CentOS 8.1, these two boxes has been rebooted serveral times previously, without any problems at all. Only after upgrading to CentOS 8.1 this happened on both of them for the first time.
I can't believe I am the only one that experience this? My guess is that this is a unknown random intermittent bug in CentOS 8.1 that kicked in. I just hope this does not continue to happen in the future. If you experienced the same, please share it with a reply. Thank you!
In order to determine what is going on you need to give a lot more information.
1. How do these boxes get their network information? DHCP or static 2. If they are static, what controls the setting of ips: NetworkManager or network-scripts 3. If they are static, how are they set in /etc/sysconfig/network-scripts/ 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY= 5. If you are using network-manager, what does nmtui or the graphical tool say the gateway or default route is?
Kind regards, Asle Ommundsen _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On Thu, 16 Jan 2020 15:34:43 +0100, Stephen John Smoogen smooge@gmail.com wrote:
On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen aommundsen@gmail.com wrote:
Hi,
Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a reboot of the first server the network was unavailable. In IPMI console everything except the network was looking good. Network was unreachable. No errors in NetworkManager. I also restarted NetworkManager, but it did not help. Then I discovered that the default gateway suddenly was missing.
Then I rebooted the server one more time, but network was still down.
Then both myself and a technician in my datcenter was debugging this (I had to wake him up in the middle of the night, costing me a lot of money), without finding any reason for why the default gateway was missing after reboot.
Then we rebooted the server a third time, and all of a sudden the problem was gone and the default gateway was back. [...cut...]
In order to determine what is going on you need to give a lot more information.
- How do these boxes get their network information? DHCP or static
- If they are static, what controls the setting of ips:
NetworkManager or network-scripts 3. If they are static, how are they set in /etc/sysconfig/network-scripts/ 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY= 5. If you are using network-manager, what does nmtui or the graphical tool say the gateway or default route is?
Here is answers to your list. I have anonymized the some of the data:
1) Static ip configuration
2) This should be NetworkManager. nmcli output:
eno1: connected to eno1 inet4 1.1.1.234/29 route4 1.1.1.232/29 route4 0.0.0.0/0
eno2: connected to eno2 inet4 192.168.0.5/24 route4 192.168.0.0/24
[root@server ~]# nmcli d show | grep IP4.GATEWA IP4.GATEWAY: 1.1.1.1.233
3) TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=eno1 UUID=1f9ec889-3c64-470a-894b-05543ee44c29 DEVICE=eno1 ONBOOT=yes IPADDR=1.1.1..234 PREFIX=29 GATEWAY=1.1.1.233 IPV6_PRIVACY=no
4) Yes
5) [root@server ~]# nmcli d show | grep IP4.GATEWA IP4.GATEWAY: 1.1.1.233
nmtui shows the same gateway.
Kind regards, Asle Ommundsen
On Thu, 16 Jan 2020 15:34:43 +0100, Stephen John Smoogen smooge@gmail.com wrote:
On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen aommundsen@gmail.com wrote:
Hi,
Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a reboot of the first server the network was unavailable. In IPMI console everything except the network was looking good. Network was unreachable. No errors in NetworkManager. I also restarted NetworkManager, but it did not help. Then I discovered that the default gateway suddenly was missing.
Then I rebooted the server one more time, but network was still down.
Then both myself and a technician in my datcenter was debugging this (I had to wake him up in the middle of the night, costing me a lot of money), without finding any reason for why the default gateway was missing after reboot.
Then we rebooted the server a third time, and all of a sudden the problem was gone and the default gateway was back. [...cut...]
In order to determine what is going on you need to give a lot more information.
- How do these boxes get their network information? DHCP or static
- If they are static, what controls the setting of ips:
NetworkManager or network-scripts 3. If they are static, how are they set in /etc/sysconfig/network-scripts/ 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY= 5. If you are using network-manager, what does nmtui or the graphical tool say the gateway or default route is?
Here is answers to your list. I have anonymized the some of the data:
Static ip configuration
This should be NetworkManager.
nmcli output:
eno1: connected to eno1 inet4 1.1.1.234/29 route4 1.1.1.232/29 route4 0.0.0.0/0
eno2: connected to eno2 inet4 192.168.0.5/24 route4 192.168.0.0/24
[root@server ~]# nmcli d show | grep IP4.GATEWA IP4.GATEWAY: 1.1.1.1.233
TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=eno1 UUID=1f9ec889-3c64-470a-894b-05543ee44c29 DEVICE=eno1 ONBOOT=yes IPADDR=1.1.1..234 PREFIX=29 GATEWAY=1.1.1.233 IPV6_PRIVACY=no
Anything in the logs about what was going on? If you reboot this server again and again, does the problem show up again?
Regards, Simon
On Fri, 17 Jan 2020 14:39:19 +0100, Simon Matter via CentOS centos@centos.org wrote:
Anything in the logs about what was going on? If you reboot this server again and again, does the problem show up again?
This is shared hosting servers that is in production with customers live sites. So I don't want to reboot again if not absolutely needed. I guess I will find out if the problem continues the next time there is a new kernel. I would like to avoid doing reboots before a new kernel is released.
I have not had time to go over the logs yet.
I was hoping that somone else experienced the same, but it does not seems so. I can't believe I would be the only one in the world this would happen to, and also that it happened on both my CentOS 8 servers.
Kind regards, Asle Ommundsen
On Fri, 17 Jan 2020 14:39:19 +0100, Simon Matter via CentOS centos@centos.org wrote:
Anything in the logs about what was going on? If you reboot this server again and again, does the problem show up again?
This is shared hosting servers that is in production with customers live sites. So I don't want to reboot again if not absolutely needed. I guess I will find out if the problem continues the next time there is a new kernel. I would like to avoid doing reboots before a new kernel is released.
I have not had time to go over the logs yet.
I was hoping that somone else experienced the same, but it does not seems so. I can't believe I would be the only one in the world this would happen to, and also that it happened on both my CentOS 8 servers.
Such things happened to me in the past as well and I couldn't believe I'm the only one. But, you have to consider certain things:
1) CentOS 8 is very new and I don't think it has already widely reached the production world. I usually don't do this before .2 comes out and I feel I'm not alone with this rule. At least it has served me well in the last decade or two.
2) The number of production servers running two different IPv4 networks on two adapters, as you do, may be quite small compared to the big number of simple settings.
3) A lot of serious production servers are still using good old init scripts to handle network settings and interfaces. Now that we're almost forced to move to the more modern approach with NetworkManager and systemd, new issues will slowly appear and get fixed step by step.
Regards, Simon
On Fri, 17 Jan 2020 at 07:37, Asle Ommundsen aommundsen@gmail.com wrote:
On Thu, 16 Jan 2020 15:34:43 +0100, Stephen John Smoogen smooge@gmail.com wrote:
On Thu, 16 Jan 2020 at 07:58, Asle Ommundsen aommundsen@gmail.com wrote:
Hi,
Tonight I upgraded two CentOS 8 boxes to CentOS 8.1 (1911). Then after a reboot of the first server the network was unavailable. In IPMI console everything except the network was looking good. Network was unreachable. No errors in NetworkManager. I also restarted NetworkManager, but it did not help. Then I discovered that the default gateway suddenly was missing.
Then I rebooted the server one more time, but network was still down.
Then both myself and a technician in my datcenter was debugging this (I had to wake him up in the middle of the night, costing me a lot of money), without finding any reason for why the default gateway was missing after reboot.
Then we rebooted the server a third time, and all of a sudden the problem was gone and the default gateway was back. [...cut...]
In order to determine what is going on you need to give a lot more information.
- How do these boxes get their network information? DHCP or static
- If they are static, what controls the setting of ips:
NetworkManager or network-scripts 3. If they are static, how are they set in /etc/sysconfig/network-scripts/ 4. Do the files in /etc/sysconfig/network-scripts list a GATEWAY= 5. If you are using network-manager, what does nmtui or the graphical tool say the gateway or default route is?
Here is answers to your list. I have anonymized the some of the data:
Thank you for the detailed answers.. they eliminated most of the 'easy-to-fix' problems. If you do not have ipv6 you might want to turn that off as I have seen some problems where a router gives enough info to cause routing issues but shouldn't have. If you do have ipv6 and it works.. then never mind.
The only other confused one i have seen is where eno1 and eno2 both have DEFROUTE=yes defined.. and you can't do that. Otherwise.. I am not sure and it will take going through the logs or repeating it happen to diagnose better.
Static ip configuration
This should be NetworkManager.
nmcli output:
eno1: connected to eno1 inet4 1.1.1.234/29 route4 1.1.1.232/29 route4 0.0.0.0/0
eno2: connected to eno2 inet4 192.168.0.5/24 route4 192.168.0.0/24
[root@server ~]# nmcli d show | grep IP4.GATEWA IP4.GATEWAY: 1.1.1.1.233
TYPE=Ethernet PROXY_METHOD=none BROWSER_ONLY=no BOOTPROTO=none DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6INIT=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no IPV6_ADDR_GEN_MODE=stable-privacy NAME=eno1 UUID=1f9ec889-3c64-470a-894b-05543ee44c29 DEVICE=eno1 ONBOOT=yes IPADDR=1.1.1..234 PREFIX=29 GATEWAY=1.1.1.233 IPV6_PRIVACY=no
Yes
[root@server ~]# nmcli d show | grep IP4.GATEWA IP4.GATEWAY: 1.1.1.233
nmtui shows the same gateway.
Kind regards, Asle Ommundsen _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
-
Asle Ommundsen -
Simon Matter -
Stephen John Smoogen