[CentOS] Server entering Emergency Shell, but continues fine after pressing Enter

Thu Sep 10 02:49:20 UTC 2020
Quinn Comendant <quinn at strangecode.com>

Hello all,

I've got an odd problem that doesn't seem to be mentioned anywhere.

I have several identical CentOS 7 servers (GCE instances). I recently ran `yum update` and rebooted all of them. All the servers came back fine except one. I opened a connection to the serial console of the broken server, and was greeted with this prompt:

    …
    Cannot open access to console, the root account is locked. 
    See sulogin(8) man page for more details. 
 
    Press Enter to continue. 

I pressed Enter, and the boot process continued successfully! Just to test, I restarted the server again, and the same thing happened: I had to manually log in to the console and press Enter before it would complete the boot process. No other step was required. 

I also tried creating a snapshot of the disk, and booted a new VM with a boot disk imaged from the snapshot. The same problem occurred, pressing Enter was all that was required. 

Earlier in the boot log, it shows "Started Emergency Shell", which is why there was a "Press Enter" prompt. The error that the "root account is locked" isn't the issue; the root account is locked on all our servers.

So, in summary: something is causing the server to enter an Emergency Shell, but continues successfully after pressing Enter. That's odd because if Emergency Shell is loaded, usually something more serious is happening and requires more actions to get the server to boot.

Anybody have any idea what could be causing this?

I don't see any significant errors in the boot log, but I would appreciate if anyone has a moment to help me look for issues. Here's a copy of the serial console boot log – you can find the "Press Enter to continue" on line 536: https://write.as/dwuts24dcw6yh0kf.txt

Thanks!

Quinn