I have a KVM vm running CentOS-6.8 on a host also running CentOS-6.8. This instance is used for occasional development projects which require segregation. Thus it is seldom accessed.
At some point in the recent past this guest developed an issue with starting. Specifically these messages were found in the system log files:
/var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_cm_xnet245.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet241.harte-lyne.ca_00' [32.00 GiB] inherit /var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet241.harte-lyne.ca_01' [32.00 GiB] inherit /var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet241.harte-lyne.ca_02' [32.00 GiB] inherit /var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet242.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet243.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet244.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages-20170312:Mar 10 16:31:06 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet245.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_cm_xnet245.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet241.harte-lyne.ca_00' [32.00 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet241.harte-lyne.ca_01' [32.00 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet241.harte-lyne.ca_02' [32.00 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet242.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet243.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet244.harte-lyne.ca_00' [31.25 GiB] inherit /var/log/messages:Mar 20 08:52:10 vhost04 kernel: dracut: inactive '/dev/vg_vhost04/lv_vm_xnet245.harte-lyne.ca_00' [31.25 GiB] inherit
It appears that this issue was first encountered on March 10 as our log files go back much further than that and have no earlier record.
The symptom presented on the guest console during boot is:
Error 13: Invalid or unsupported executable format
Press any key to continue...
Pressing enter brings up the following text display
GNU GRUB version 0.97 (615K lower / 3668980K upper memory)
CentOS (2.6.32-641.15.1.el6.x86_64) CentOS (2.6.32-641.13.1.el6.x86_64) CentOS (2.6.32-641.11.1.el6.x86_64) CentOS (2.6.32-641.2.1.el6.x86_64)
Use the ↑ and ↓ keys to select which entry is highlighted Press enter to boot the selected OS, 'e' to edit . . .
The first choice repeats the Error. The second choice boots cleanly.
Looking at yum history I see this:
yum history Loaded plugins: etckeeper, fastestmirror, priorities, refresh-packagekit, security ID | Login user | Date and time | Action(s) | Altered --------------------------------------------------------------------- . . . 368 | root <root> | 2017-03-10 16:46 | Update | 4 367 | root <root> | 2017-03-10 16:42 | E, U | 35 EE . . .
Looking at transaction 367 more closely we see that the kernel was updated to 2.6.32-642.15.1.el6.x86_64 on March 10 but that a number or errors, whose nature I do not comprehend, were also reported.
# yum history info 367 Loaded plugins: etckeeper, fastestmirror, priorities, refresh-packagekit, security Transaction ID : 367 Begin time : Fri Mar 10 16:42:32 2017 Begin rpmdb : 1489:fd0eb9a01b1667f826b8fead9bc0a05e5bc43efd End time : 16:43:59 2017 (87 seconds) End rpmdb : 1461:cac690d6280fa97910ccb59d0d1f6d43990dfd0a User : root <root> Return-Code : Success Transaction performed with: Installed rpm-4.8.0-55.el6.x86_64 @base Installed yum-3.2.29-75.el6.centos.noarch @updates Installed yum-metadata-parser-1.1.2-16.el6.x86_64 @anaconda-CentOS-201207061011.x86_64/6.3 Installed yum-plugin-fastestmirror-1.1.30-37.el6.noarch @base Installed yum-utils-1.1.30-37.el6.noarch @base Packages Altered: Updated firefox-45.7.0-1.el6.centos.x86_64 @updates Update 45.7.0-2.el6.centos.x86_64 @updates Updated gnome-settings-daemon-2.28.2-35.el6.x86_64 @base Update 2.28.2-35.el6_8.2.x86_64 @updates Erase initscripts-9.03.53-1.el6.centos.1.x86_64 @updates Erase kernel-2.6.32-642.4.2.el6.x86_64 @updates Erase kernel-firmware-2.6.32-642.13.1.el6.noarch @updates Updated kernel-headers-2.6.32-642.13.1.el6.x86_64 @updates Update 2.6.32-642.15.1.el6.x86_64 @updates Updated kexec-tools-2.0.0-300.el6_8.1.x86_64 @updates Update 2.0.0-300.el6_8.2.x86_64 @updates Erase libbasicobjects-0.1.1-11.el6.x86_64 @base Erase libblkid-2.17.2-12.24.el6_8.1.x86_64 @updates Erase libcollection-0.6.2-11.el6.x86_64 @base Erase libdhash-0.4.3-11.el6.x86_64 @base Erase libini_config-1.1.0-11.el6.x86_64 @base Erase libipa_hbac-1.13.3-22.el6_8.4.x86_64 @updates Erase libpath_utils-0.2.1-11.el6.x86_64 @base Erase libref_array-0.1.4-11.el6.x86_64 @base Erase libsss_idmap-1.13.3-22.el6_8.4.x86_64 @updates Erase libuuid-2.17.2-12.24.el6_8.1.x86_64 @updates Updated openssl-1.0.1e-48.el6_8.3.i686 @updates Erase openssl-1.0.1e-48.el6_8.3.x86_64 @updates Update openssl-1.0.1e-48.el6_8.4.i686 @updates Updated openssl-devel-1.0.1e-48.el6_8.3.x86_64 @updates Update 1.0.1e-48.el6_8.4.x86_64 @updates Updated python-libipa_hbac-1.13.3-22.el6_8.4.x86_64 @updates Update 1.13.3-22.el6_8.6.x86_64 @updates Erase python-sssdconfig-1.13.3-22.el6_8.4.noarch @updates Erase selinux-policy-3.7.19-292.el6_8.2.noarch @updates Erase selinux-policy-targeted-3.7.19-292.el6_8.2.noarch @updates Erase sssd-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-ad-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-client-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-common-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-common-pac-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-ipa-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-krb5-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-krb5-common-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-ldap-1.13.3-22.el6_8.4.x86_64 @updates Erase sssd-proxy-1.13.3-22.el6_8.4.x86_64 @updates Erase util-linux-ng-2.17.2-12.24.el6_8.1.x86_64 @updates Scriptlet output: 1 warning: erase unlink of /lib/modules/2.6.32-642.4.2.el6.x86_64/weak-updates failed: No such file or directory 2 warning: erase unlink of /lib/modules/2.6.32-642.4.2.el6.x86_64/modules.order failed: No such file or directory 3 warning: erase unlink of /lib/modules/2.6.32-642.4.2.el6.x86_64/modules.networking failed: No such file or directory 4 warning: erase unlink of /lib/modules/2.6.32-642.4.2.el6.x86_64/modules.modesetting failed: No such file or directory 5 warning: erase unlink of /lib/modules/2.6.32-642.4.2.el6.x86_64/modules.drm failed: No such file or directory 6 warning: erase unlink of /lib/modules/2.6.32-642.4.2.el6.x86_64/modules.block failed: No such file or directory
Reviewing grub.conf I note that the initrd entry is missing from the newest kernel entry. Which of course explains the boot error message. However, as kernel updates are entirely automated when performed by yum I am at a lost to explain its absence.
default=saved fallback= 0 1 2 timeout=5 splashimage=(hd0,0)/grub/splash.xpm.gz hiddenmenu title CentOS (2.6.32-642.15.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-642.15.1.el6.x86_64 ro root=/dev/mapper/vg_xnet241-lv_root rd_NO_LUKS LANG=en_CA@yyyy-mmm-dd.UTF-8 rd_LVM_LV=vg_xnet241/lv_swap rd_LVM_LV=vg_xnet241/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto radeon.modeset=0 title CentOS (2.6.32-642.13.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-642.13.1.el6.x86_64 ro root=/dev/mapper/vg_xnet241-lv_root rd_NO_LUKS LANG=en_CA@yyyy-mmm-dd.UTF-8 rd_LVM_LV=vg_xnet241/lv_swap rd_LVM_LV=vg_xnet241/lv_root rd_NO_MD SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto radeon.modeset=0 initrd /initramfs-2.6.32-642.13.1.el6.x86_64.img
There is no initramfs-2.6.32-642.15.1.el6.x86_64.img file present in /boot so something is seriously wrong somewhere.
As shown above I have since reconfigured grub.conf to fallback to the last working kernel. Nonetheless, I would like guidance on how to proceed with fixing/removing the broken one.
Suggestions?