I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
I have been punished by this bug - it is/was very nasty.
I will chase up your reference but attach my trouble shooting in case its useful to anyone
John
Il 31/07/20 13:08, ja ha scritto:
On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
I have been punished by this bug - it is/was very nasty.
Me too. Luckily it happened on a test machine.
Sorry but seems that those packages were not tested before pushing them in the update repo. Would be great to know what happened to the mainstream chains and how a package like grub reached the update repo when it has serious problem (genuine curiosity but not to blame them).
In my case, the restore procedure reported by RH in case of reboot does not work and reports that the packages are already at the lowest version and that the downgrade is not possibile. I don't know why.
RedHat is recommending not to patch. https://access.redhat.com/solutions/5272311
Andrea
-----Original Message----- From: CentOS centos-bounces@centos.org On Behalf Of Alessandro Baggi Sent: Friday, July 31, 2020 11:25 AM To: centos@centos.org Subject: {EXTERNAL} Re: [CentOS] 8.2.2004 Latest yum update renders machine unbootable
CAUTION: This email originated outside of BSWH; avoid action unless you know the content is safe. Send suspicious emails as attachments to BadEmail@BSWHealth.org.
Il 31/07/20 13:08, ja ha scritto:
On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
I have been punished by this bug - it is/was very nasty.
Me too. Luckily it happened on a test machine.
Sorry but seems that those packages were not tested before pushing them in the update repo. Would be great to know what happened to the mainstream chains and how a package like grub reached the update repo when it has serious problem (genuine curiosity but not to blame them).
In my case, the restore procedure reported by RH in case of reboot does not work and reports that the packages are already at the lowest version and that the downgrade is not possibile. I don't know why.
_______________________________________________ CentOS mailing list CentOS@centos.org https://urldefense.com/v3/__https://lists.centos.org/mailman/listinfo/centos...
********************************************************************** The information contained in this e-mail may be privileged and/or confidential, and protected from disclosure, and no waiver of any attorney-client, work product, or other privilege is intended. If you are the intended recipient, further disclosures are prohibited without proper authorization. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden and possibly a violation of federal or state law and regulations. The sender and Baylor Scott & White Health, and its affiliated entities, hereby expressly reserve all privileges and confidentiality that might otherwise be waived as a result of an erroneous or misdirected e-mail transmission. No employee or agent is authorized to conclude any binding agreement on behalf of Baylor Scott & White Health, or any affiliated entity, by e-mail without express written confirmation by the CEO, the Senior Vice President of Supply Chain Services or other duly authorized representative of Baylor Scott & White Health.
I would add,
RedHat is recomending also to blacklist grub, shim and mokutil in yum.conf until the problem is resolved.
Il 31/07/20 18:26, Laack, Andrea P ha scritto:
RedHat is recommending not to patch. https://access.redhat.com/solutions/5272311
Andrea
-----Original Message----- From: CentOS centos-bounces@centos.org On Behalf Of Alessandro Baggi Sent: Friday, July 31, 2020 11:25 AM To: centos@centos.org Subject: {EXTERNAL} Re: [CentOS] 8.2.2004 Latest yum update renders machine unbootable
CAUTION: This email originated outside of BSWH; avoid action unless you know the content is safe. Send suspicious emails as attachments to BadEmail@BSWHealth.org.
Il 31/07/20 13:08, ja ha scritto:
On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
I have been punished by this bug - it is/was very nasty.
Me too. Luckily it happened on a test machine.
Sorry but seems that those packages were not tested before pushing them in the update repo. Would be great to know what happened to the mainstream chains and how a package like grub reached the update repo when it has serious problem (genuine curiosity but not to blame them).
In my case, the restore procedure reported by RH in case of reboot does not work and reports that the packages are already at the lowest version and that the downgrade is not possibile. I don't know why.
CentOS mailing list CentOS@centos.org https://urldefense.com/v3/__https://lists.centos.org/mailman/listinfo/centos...
The information contained in this e-mail may be privileged and/or confidential, and protected from disclosure, and no waiver of any attorney-client, work product, or other privilege is intended. If you are the intended recipient, further disclosures are prohibited without proper authorization. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized copying, disclosure or distribution of the material in this e-mail is strictly forbidden and possibly a violation of federal or state law and regulations. The sender and Baylor Scott & White Health, and its affiliated entities, hereby expressly reserve all privileges and confidentiality that might otherwise be waived as a result of an erroneous or misdirected e-mail transmission. No employee or agent is authorized to conclude any binding agreement on behalf of Baylor Scott & White Health, or any affiliated entity, by e-mail without express written confirmation by the CEO, the Senior Vice President of Supply Chain Services or other duly authorized representative of Baylor Scott & White Health. _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 7/31/20 11:24 AM, Alessandro Baggi wrote:
Il 31/07/20 13:08, ja ha scritto:
On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
I have been punished by this bug - it is/was very nasty.
Me too. Luckily it happened on a test machine.
Sorry but seems that those packages were not tested before pushing them in the update repo. Would be great to know what happened to the mainstream chains and how a package like grub reached the update repo when it has serious problem (genuine curiosity but not to blame them).
Of course it was tested before it was pushed. Obviously this is not a problem with every install. Surely you don't think we push items without doing any testing. Certainly not items as important as this update.
The CentOS infrastructure has hundreds of servers, most of them were not impacted (as an example). In fact, we seem to have had this happen on only one machine in those hundreds so far. It is a problem, obviously.
The issue seems to be with the shim package (not the grub or kernel packages) and we are currently working with Red Hat on a fix. This issue happened in many Linux OSes and even Windows, not just RHEL and CentOS.
We will push a fix as soon as one is available.
I would hold off on installing this until we release the new fixes.
In my case, the restore procedure reported by RH in case of reboot does not work and reports that the packages are already at the lowest version and that the downgrade is not possibile. I don't know why.
On 7/31/20 11:20 PM, Johnny Hughes wrote:
On 7/31/20 11:24 AM, Alessandro Baggi wrote:
Il 31/07/20 13:08, ja ha scritto:
On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
I have been punished by this bug - it is/was very nasty.
Me too. Luckily it happened on a test machine.
Sorry but seems that those packages were not tested before pushing them in the update repo. Would be great to know what happened to the mainstream chains and how a package like grub reached the update repo when it has serious problem (genuine curiosity but not to blame them).
Of course it was tested before it was pushed. Obviously this is not a problem with every install. Surely you don't think we push items without doing any testing. Certainly not items as important as this update.
The CentOS infrastructure has hundreds of servers, most of them were not impacted (as an example). In fact, we seem to have had this happen on only one machine in those hundreds so far. It is a problem, obviously.
The issue seems to be with the shim package (not the grub or kernel packages) and we are currently working with Red Hat on a fix. This issue happened in many Linux OSes and even Windows, not just RHEL and CentOS.
We will push a fix as soon as one is available.
I would hold off on installing this until we release the new fixes.
In my case, the restore procedure reported by RH in case of reboot does not work and reports that the packages are already at the lowest version and that the downgrade is not possibile. I don't know why.
It tanked my machine. I managed to get it to boot into emergency mode where I eventually got it to boot on an old kernel as shown in the paste below. After getting it to boot I used grubby to set the default kernel to the oldest one on the machine. Surprisingly enough neither of the two most recent kernels would boot even though the machine was running on the second oldest kernel when I ran the update that tanked it. That adds credence to the comment above that the problem was not the new kernel since it trashed both the new kernel and the one before. Both were from the same series. The 147 kernel works but neither of newer ones would.
CentOS Linux release 8.2.2004 (Core)
enp5s0 IP Address = 192.168.15.131
Linux localhost.localdomain 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Number of cores = 32 00:55:13 up 8:37, 1 user, load average: 0.31, 0.48, 0.44
amdgpu-pci-0900 Adapter: PCI adapter vddgfx: +0.83 V fan1: 771 RPM (min = 0 RPM, max = 3700 RPM) temp1: +45.0°C (crit = +94.0°C, hyst = -273.1°C) power1: 31.07 W (cap = 125.00 W)
acpitz-virtual-0 Adapter: Virtual device temp1: +16.8°C (crit = +20.8°C) temp2: +16.8°C (crit = +20.8°C)
amdgpu-pci-0400 Adapter: PCI adapter vddgfx: +0.72 V fan1: 768 RPM (min = 0 RPM, max = 3700 RPM) temp1: +39.0°C (crit = +94.0°C, hyst = -273.1°C) power1: 30.10 W (cap = 125.00 W)
On Fri, Jul 31, 2020 at 10:20:28PM -0500, Johnny Hughes wrote: ...
The issue seems to be with the shim package (not the grub or kernel packages) and we are currently working with Red Hat on a fix. This issue happened in many Linux OSes and even Windows, not just RHEL and CentOS.
We will push a fix as soon as one is available.
I would hold off on installing this until we release the new fixes.
On a system that does not use any of the shim packages, would you recommend updating the grub2 and mokutil packages or holding off?
Jon
On 8/1/20 1:59 AM, Jon LaBadie wrote:
On Fri, Jul 31, 2020 at 10:20:28PM -0500, Johnny Hughes wrote: ...
The issue seems to be with the shim package (not the grub or kernel packages) and we are currently working with Red Hat on a fix. This issue happened in many Linux OSes and even Windows, not just RHEL and CentOS.
We will push a fix as soon as one is available.
I would hold off on installing this until we release the new fixes.
On a system that does not use any of the shim packages, would you recommend updating the grub2 and mokutil packages or holding off?
Jon
I would wait and install everything as a group. We should have something soon.
On 8/1/20 5:00 AM, Johnny Hughes wrote:
I would wait and install everything as a group. We should have something soon.
First off, Johnny and all of the rest of the CentOS team, thank you for your efforts!
Second, to all with this problem, I too experienced the issue (I posted on the CentOS-Devel list my findings). To those who seem to think more testing could have prevented the issue, let me just say that while the FIRST update to the new shim-x64, grub2, and kernel stack failed for me (and that update was done with the GUI updater that reboots for the actual update), I WAS able to recover using the install media's rescue mode tools as documented in my posts on CentOS-devel. I'm glad to provide any and all logs of the processes I did, but the short of it is that after I successfully downgraded grub2 it still wouldn't boot, but I was able to do a reinstall of grub2 (81, the older version) to get it to boot with the old grub2, the new shim-x64, and the new kernel, and then I was able to successfully update grub2 with dnf on a command line. So I have the following installed: [lowen@localhost ~]$ rpm -q shim-x64 shim-x64-15-13.el8.x86_64 [lowen@localhost ~]$ rpm -qa | grep ^grub2 grub2-common-2.02-87.el8_2.noarch grub2-tools-2.02-87.el8_2.x86_64 grub2-efi-x64-2.02-87.el8_2.x86_64 grub2-tools-extra-2.02-87.el8_2.x86_64 grub2-pc-2.02-87.el8_2.x86_64 grub2-tools-efi-2.02-87.el8_2.x86_64 grub2-tools-minimal-2.02-87.el8_2.x86_64 grub2-pc-modules-2.02-87.el8_2.noarch grub2-efi-x64-modules-2.02-87.el8_2.noarch [lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 147 kernel-devel-4.18.0-147.8.1.el8_1.x86_64 kernel-4.18.0-147.8.1.el8_1.x86_64 kernel-modules-4.18.0-147.8.1.el8_1.x86_64 kernel-core-4.18.0-147.8.1.el8_1.x86_64 [lowen@localhost ~]$
And this is on a machine that previously crashed during boot with the exactly the same packages. So even on the same hardware the same update (done a bit differently) had different results.
It's quite possible, based on my experience with this particular issue, that if I had the whole system rolled back to where it was before the GUI update and if I then updated it again that it could just simply work ok afterwards. It's very hard to test for intermittent issues!
On 8/1/20 11:02 AM, Lamar Owen wrote:
... [lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 147 kernel-devel-4.18.0-147.8.1.el8_1.x86_64 kernel-4.18.0-147.8.1.el8_1.x86_64 kernel-modules-4.18.0-147.8.1.el8_1.x86_64 kernel-core-4.18.0-147.8.1.el8_1.x86_64 [lowen@localhost ~]$
Well, I sure fat-fingered that command.... let's try it again:
[lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 193.14 kernel-headers-4.18.0-193.14.2.el8_2.x86_64 kernel-devel-4.18.0-193.14.2.el8_2.x86_64 kernel-modules-4.18.0-193.14.2.el8_2.x86_64 kernel-tools-4.18.0-193.14.2.el8_2.x86_64 kernel-core-4.18.0-193.14.2.el8_2.x86_64 kernel-tools-libs-4.18.0-193.14.2.el8_2.x86_64 kernel-4.18.0-193.14.2.el8_2.x86_64 [lowen@localhost ~]$ uname -a Linux localhost.localdomain 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [lowen@localhost ~]$
On Sat, Aug 1, 2020 at 8:58 PM Lamar Owen lowen@pari.edu wrote:
On 8/1/20 11:02 AM, Lamar Owen wrote:
... [lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 147 kernel-devel-4.18.0-147.8.1.el8_1.x86_64 kernel-4.18.0-147.8.1.el8_1.x86_64 kernel-modules-4.18.0-147.8.1.el8_1.x86_64 kernel-core-4.18.0-147.8.1.el8_1.x86_64 [lowen@localhost ~]$
Well, I sure fat-fingered that command.... let's try it again:
[lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 193.14 kernel-headers-4.18.0-193.14.2.el8_2.x86_64 kernel-devel-4.18.0-193.14.2.el8_2.x86_64 kernel-modules-4.18.0-193.14.2.el8_2.x86_64 kernel-tools-4.18.0-193.14.2.el8_2.x86_64 kernel-core-4.18.0-193.14.2.el8_2.x86_64 kernel-tools-libs-4.18.0-193.14.2.el8_2.x86_64 kernel-4.18.0-193.14.2.el8_2.x86_64 [lowen@localhost ~]$ uname -a Linux localhost.localdomain 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul 26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [lowen@localhost ~]$
RedHat fixes.
https://access.redhat.com/errata/RHBA-2020:3265 https://access.redhat.com/errata/RHBA-2020:3262
-- Lee
I got 5 CentOS 8.1 VMs minimal install. When I did the update I got 1 that had dnf issues and corrupted rpm DB and another 1 that didn't boot. Restore from snapshot and again update -> no issues. For me it looks more like a random issue.
Best Regards, Strahil Nikolov
На 1 август 2020 г. 18:28:09 GMT+03:00, Lamar Owen lowen@pari.edu написа:
On 8/1/20 11:02 AM, Lamar Owen wrote:
... [lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 147 kernel-devel-4.18.0-147.8.1.el8_1.x86_64 kernel-4.18.0-147.8.1.el8_1.x86_64 kernel-modules-4.18.0-147.8.1.el8_1.x86_64 kernel-core-4.18.0-147.8.1.el8_1.x86_64 [lowen@localhost ~]$
Well, I sure fat-fingered that command.... let's try it again:
[lowen@localhost ~]$ rpm -qa | grep ^kernel|grep 193.14 kernel-headers-4.18.0-193.14.2.el8_2.x86_64 kernel-devel-4.18.0-193.14.2.el8_2.x86_64 kernel-modules-4.18.0-193.14.2.el8_2.x86_64 kernel-tools-4.18.0-193.14.2.el8_2.x86_64 kernel-core-4.18.0-193.14.2.el8_2.x86_64 kernel-tools-libs-4.18.0-193.14.2.el8_2.x86_64 kernel-4.18.0-193.14.2.el8_2.x86_64 [lowen@localhost ~]$ uname -a Linux localhost.localdomain 4.18.0-193.14.2.el8_2.x86_64 #1 SMP Sun Jul
26 03:54:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux [lowen@localhost ~]$
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
Hi Johnny, thank you very much for clarification.
You said that in the centos infrastructure only one server got the problem. What are the conditions that permit the breakage? There is a particular configuration (hw/sw) case that match always the problem or it is random?
Thank you
Il Sab 1 Ago 2020, 05:20 Johnny Hughes johnny@centos.org ha scritto:
On 7/31/20 11:24 AM, Alessandro Baggi wrote:
Il 31/07/20 13:08, ja ha scritto:
On Fri, 2020-07-31 at 22:35 +1200, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
After some hours I managed to modify another bootable partition (containing older software) and boot it from there.
After that, I found out it is a known problem.
The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
See 'UEFI boot blank screen post update' for a solution and directions to the redhat article.
Regards
Alan
I have been punished by this bug - it is/was very nasty.
Me too. Luckily it happened on a test machine.
Sorry but seems that those packages were not tested before pushing them in the update repo. Would be great to know what happened to the mainstream chains and how a package like grub reached the update repo when it has serious problem (genuine curiosity but not to blame them).
Of course it was tested before it was pushed. Obviously this is not a problem with every install. Surely you don't think we push items without doing any testing. Certainly not items as important as this update.
The CentOS infrastructure has hundreds of servers, most of them were not impacted (as an example). In fact, we seem to have had this happen on only one machine in those hundreds so far. It is a problem, obviously.
The issue seems to be with the shim package (not the grub or kernel packages) and we are currently working with Red Hat on a fix. This issue happened in many Linux OSes and even Windows, not just RHEL and CentOS.
We will push a fix as soon as one is available.
I would hold off on installing this until we release the new fixes.
In my case, the restore procedure reported by RH in case of reboot does not work and reports that the packages are already at the lowest version and that the downgrade is not possibile. I don't know why.
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 8/1/20 5:54 AM, Alessandro Baggi wrote:
You said that in the centos infrastructure only one server got the problem. What are the conditions that permit the breakage? There is a particular configuration (hw/sw) case that match always the problem or it is random?
I experienced the issue on a Dell Precision M6700 (UEFI, Core i7-3740QM) on the first attempt at update; I recovered by rolling back and reinstalling grub2-*; I then updated on the command line with dnf update and now that same hardware that crashed out the first time is working fine with the same package versions. It seems to be intermittent, and I can't reproduce it.
On 7/31/20 6:35 AM, Alan McRae via CentOS wrote:
I am running an Intel x64 machine using UEFI to boot an SSD.
Installing the latest yum update which includes grub2 and kernel 4.18.0-193.14.2.el8_2.x86_64 renders the machine unbootable, blank screen where grub should be, no error messages, just hangs.
... The main point of this message is to make people aware of the problem and suggest admins don't run 'yum update' until they understand the problem and have a fix at hand.
I posted, on 7/30, to CentOS-devel the following after my similar experience (booting an mSATA SSD on my M6700, with two 2.5 SATA drives (a 1TB and a 2TB), and the mSATA is not /dev/sda on this hardware, FWIW):
"Moral of the story? Have the install DVD on a USB stick and available to boot into rescue mode, know how to boot rescue mode (from the installer boot menu select 'troubleshooting....' then 'rescue....' and option 1 once the menu shows), know how to activate network interfaces in text mode (either nmcli or nmtui works fine for this), and know a few basic dnf commands. And have an alternate means of doing basic web searches available; my android phone this morning was used for that....."
I was able to recover with the install DVD on a USB stick and using rescue mode on a laptop with only WiFi (nmtui does a great job for this sort of thing, booted into rescue mode and chrooted onto /mnt/sysimage).