Ran a 'yum update' on my Banana Pi firewall this morning and ended up with an unbootable system. (The previous kernel wouldn't boot either.)
I ended up copying my entire SD card, installing the latest image, growing the / partition, and copying the old root filesystem over. Ultimately, I was able to get everything up and running.
This system was originally installed with CentOS-Userland-7-armv7hl-Minimal-1611-BananaPi.img.xz.
I'm mainly wondering if this was expected. If so, did I miss the warnings? If not, I do still have the dump of the unbootable post- upgrade SD card sitting around, if anyone who understands the boot process on these things wants to investigate.
Either way, thanks for all the work that you all do!
Ian, that is not expected at all, especially not booting with the old kernel. Since you still have the old contents, can you paste the contents of /boot/extlinux/extlinux.conf? BTW, which BananaPi do you use? I've updated all my BPi-M1 without issues, but it is a rule for me to update in this order: 1) yum and rpm 2) all but kernel 3) kernel To maybe that is why it didn't happen to me.
Thanks. Pablo.
El 14/5/18 a las 18:36, Ian Pilcher escribió:
Ran a 'yum update' on my Banana Pi firewall this morning and ended up with an unbootable system. (The previous kernel wouldn't boot either.)
I ended up copying my entire SD card, installing the latest image, growing the / partition, and copying the old root filesystem over. Ultimately, I was able to get everything up and running.
This system was originally installed with CentOS-Userland-7-armv7hl-Minimal-1611-BananaPi.img.xz.
I'm mainly wondering if this was expected. If so, did I miss the warnings? If not, I do still have the dump of the unbootable post- upgrade SD card sitting around, if anyone who understands the boot process on these things wants to investigate.
Either way, thanks for all the work that you all do!
On 05/14/2018 07:18 PM, Pablo Sebastián Greco wrote:
Ian, that is not expected at all, especially not booting with the old kernel. Since you still have the old contents, can you paste the contents of /boot/extlinux/extlinux.conf?
Sure. I made a few changes to try to get the system to boot, but I'm 99% sure that this is what I had immediately after the 'yum update'.
#Created by RootFS Build Factory ui menu.c32 menu autoboot centos menu title centos Options #menu hidden timeout 60 totaltimeout 600
default=centos label CentOS Linux (4.14.28-201.el7.centos.armv7hl) 7 (Core) kernel /vmlinuz-4.14.28-201.el7.centos.armv7hl append root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e LANG=en_US.UTF-8 initrd /initramfs-4.14.28-201.el7.centos.armv7hl.img
label centos kernel /vmlinuz-4.9.75-204.el7.centos.armv7hl append root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e debug initrd /initramfs-4.9.75-204.el7.centos.armv7hl.img
The thing that jumps out is that the fdt/fdtdir entries are missing.
I actually did try adding an fdt line, but I missed the fact that it's a directory, so that didn't work. It's entirely possible that adding the correct fdtdir line would have worked.
BTW, which BananaPi do you use? I've updated all my BPi-M1 without issues, but it is a rule for me to update in this order:
- yum and rpm
- all but kernel
- kernel
To maybe that is why it didn't happen to me.
I believe that it's an M1, but I'm not 100% sure how to tell. (It's definitely a dual-core ARMv7 with 1GB.)
Ian, that is definitely one of the things that changed, and executing extlinux should fix your issue. Maybe there is a timing problem between the installation of extlinux-bootloader and the new kernel, but it should only affect those who didn't have exlinux-bootloader installed. Please add fdtdir /dtb-4.14.28-201.el7.centos.armv7hl and fdtdir /dtb-4.9.75-204.el7.centos.armv7hl to both kernels respectively. If that works, I'll try to add a note to with some recommendations.
Thanks. Pablo.
El 15/5/18 a las 11:23, Ian Pilcher escribió:
On 05/14/2018 07:18 PM, Pablo Sebastián Greco wrote:
Ian, that is not expected at all, especially not booting with the old kernel. Since you still have the old contents, can you paste the contents of /boot/extlinux/extlinux.conf?
Sure. I made a few changes to try to get the system to boot, but I'm 99% sure that this is what I had immediately after the 'yum update'.
#Created by RootFS Build Factory ui menu.c32 menu autoboot centos menu title centos Options #menu hidden timeout 60 totaltimeout 600
default=centos label CentOS Linux (4.14.28-201.el7.centos.armv7hl) 7 (Core) kernel /vmlinuz-4.14.28-201.el7.centos.armv7hl append root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e LANG=en_US.UTF-8 initrd /initramfs-4.14.28-201.el7.centos.armv7hl.img
label centos kernel /vmlinuz-4.9.75-204.el7.centos.armv7hl append root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e debug initrd /initramfs-4.9.75-204.el7.centos.armv7hl.img
The thing that jumps out is that the fdt/fdtdir entries are missing.
I actually did try adding an fdt line, but I missed the fact that it's a directory, so that didn't work. It's entirely possible that adding the correct fdtdir line would have worked.
BTW, which BananaPi do you use? I've updated all my BPi-M1 without issues, but it is a rule for me to update in this order:
- yum and rpm
- all but kernel
- kernel
To maybe that is why it didn't happen to me.
I believe that it's an M1, but I'm not 100% sure how to tell. (It's definitely a dual-core ARMv7 with 1GB.)
On 05/15/2018 10:17 AM, Pablo Sebastián Greco wrote:
Ian, that is definitely one of the things that changed, and executing extlinux should fix your issue. Maybe there is a timing problem between the installation of extlinux-bootloader and the new kernel, but it should only affect those who didn't have exlinux-bootloader installed.
Looks like it wasn't previously installed:
May 14 10:39:27 Installed: extlinux-bootloader-1.2-3.el7.armv7hl
Please add fdtdir /dtb-4.14.28-201.el7.centos.armv7hl and fdtdir /dtb-4.9.75-204.el7.centos.armv7hl to both kernels respectively. If that works, I'll try to add a note to with some recommendations.
I would need to re-image the SD card to test that, which would mean taking the house completely offline (Internet & phones) for however long it takes to test and/or recover. Needless to say, I'm not eager to do that.
Is it possible to test this with QEMU somehow?
El 15/5/18 a las 17:24, Ian Pilcher escribió:
On 05/15/2018 10:17 AM, Pablo Sebastián Greco wrote:
Ian, that is definitely one of the things that changed, and executing extlinux should fix your issue. Maybe there is a timing problem between the installation of extlinux-bootloader and the new kernel, but it should only affect those who didn't have exlinux-bootloader installed.
Looks like it wasn't previously installed:
May 14 10:39:27 Installed: extlinux-bootloader-1.2-3.el7.armv7hl
Ok, I'll try to document this so it doesn't happen again
Please add fdtdir /dtb-4.14.28-201.el7.centos.armv7hl and fdtdir /dtb-4.9.75-204.el7.centos.armv7hl to both kernels respectively. If that works, I'll try to add a note to with some recommendations.
I would need to re-image the SD card to test that, which would mean taking the house completely offline (Internet & phones) for however long it takes to test and/or recover. Needless to say, I'm not eager to do that.
I understand, I wouldn't want to take everything offline just to test either.
Is it possible to test this with QEMU somehow?
Not really, the boot sequence is completely different.
If you manage to test using other device, please let us know. I'm pretty confident on what the problem was, and I'll see how to add that to the release notes.
Thanks. Pablo.
I just did my first 7.5 kernel upgrade, and I'm happy to report that the system came right up into the new kernel (the first time that it's ever done so without manual edits to extlinux.conf).
In fact, I was a bit surprised that it booted the *new* kernel. Here is my extlinux.conf:
# extlinux.conf generated by appliance-creator ui menu.c32 menu autoboot Welcome to CentOS-Userland-7-armv7hl-generic-Minimal-1804. Automatic boot in # second{,s}. Press a key for options. menu title CentOS-Userland-7-armv7hl-generic-Minimal-1804 Boot Options. menu hidden timeout 20 totaltimeout 600
default=CentOS-Userland-7-armv7hl-generic-Minimal-1804 (4.14.28-201.el7.centos.armv7hl) label CentOS Linux (4.14.43-201.el7.armv7hl) 7 (Core) kernel /vmlinuz-4.14.43-201.el7.armv7hl append ro root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e LANG=en_US.UTF-8 fdtdir /dtb-4.14.43-201.el7.armv7hl initrd /initramfs-4.14.43-201.el7.armv7hl.img
label CentOS-Userland-7-armv7hl-generic-Minimal-1804 (4.14.28-201.el7.centos.armv7hl) kernel /vmlinuz-4.14.28-201.el7.centos.armv7hl append ro root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e fdtdir /dtb-4.14.28-201.el7.centos.armv7hl initrd /initramfs-4.14.28-201.el7.centos.armv7hl.img
Based on the "default=..." line, I expected it to boot 4.14.28-201.el7.centos.armv7hl. Instead it booted into 4.14.43-201.el7.armv7hl.
So something still seems just a bit off.
El 31/5/18 a las 12:25, Ian Pilcher escribió:
I just did my first 7.5 kernel upgrade, and I'm happy to report that the system came right up into the new kernel (the first time that it's ever done so without manual edits to extlinux.conf).
There used to be a script (updateboot), but we obsoleted that one for 7.5
In fact, I was a bit surprised that it booted the *new* kernel. Here is my extlinux.conf:
# extlinux.conf generated by appliance-creator ui menu.c32 menu autoboot Welcome to CentOS-Userland-7-armv7hl-generic-Minimal-1804. Automatic boot in # second{,s}. Press a key for options. menu title CentOS-Userland-7-armv7hl-generic-Minimal-1804 Boot Options. menu hidden timeout 20 totaltimeout 600
default=CentOS-Userland-7-armv7hl-generic-Minimal-1804 (4.14.28-201.el7.centos.armv7hl) label CentOS Linux (4.14.43-201.el7.armv7hl) 7 (Core) kernel /vmlinuz-4.14.43-201.el7.armv7hl append ro root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e LANG=en_US.UTF-8 fdtdir /dtb-4.14.43-201.el7.armv7hl initrd /initramfs-4.14.43-201.el7.armv7hl.img
label CentOS-Userland-7-armv7hl-generic-Minimal-1804 (4.14.28-201.el7.centos.armv7hl) kernel /vmlinuz-4.14.28-201.el7.centos.armv7hl append ro root=UUID=eb2c92c6-69cd-4a87-982d-900be79e928e fdtdir /dtb-4.14.28-201.el7.centos.armv7hl initrd /initramfs-4.14.28-201.el7.centos.armv7hl.img
Based on the "default=..." line, I expected it to boot 4.14.28-201.el7.centos.armv7hl. Instead it booted into 4.14.43-201.el7.armv7hl.
Well, that part is simple, default=... is ignored by uboot (if you look while booting, you'll see Ignoring unknown command: default=centos)
So something still seems just a bit off.
Agreed, either that line should be deleted, or up to date.
Thanks. Pablo.
-
Ian Pilcher -
Pablo Sebastián Greco