We have Xeon IA32 dual-processor servers running Centos 3.5 in an HPC batch-only compute grid configuration . We have yum update operating automatically with default updates being applied weekly. Because of the workload pattern of long-runing jobs, the servers tend to stay up without a reboot for very long periods.
Recently, yum installed an updated kernel 2.4.21-32.0.1.ELsmp; when we got around to rebooting, we found that some of the machines were running the uniprocessor kernel 2.4.21-32.0.1.EL , showing only a single cpu. The grub.conf file had been modified in the usual pushdown manner but the default kernel had been set at #2 instead of #0.
Bizzarely, some of the systems DID boot the upgraded SMP kernel as expected.
Here is the grub.conf from an affected server:
default=2 timeout=10 splashimage=(hd0,0)/grub/splash.xpm.gz title CentOS (2.4.21-32.0.1.ELsmp) root (hd0,0) kernel /vmlinuz-2.4.21-32.0.1.ELsmp ro root=LABEL=/ initrd /initrd-2.4.21-32.0.1.ELsmp.img title CentOS (2.4.21-32.0.1.EL) root (hd0,0) kernel /vmlinuz-2.4.21-32.0.1.EL ro root=LABEL=/ initrd /initrd-2.4.21-32.0.1.EL.img title CentOS-3 (2.4.21-32.ELsmp) root (hd0,0) kernel /vmlinuz-2.4.21-32.ELsmp ro root=LABEL=/ initrd /initrd-2.4.21-32.ELsmp.img title CentOS-3-up (2.4.21-32.EL) root (hd0,0) kernel /vmlinuz-2.4.21-32.EL ro root=LABEL=/ initrd /initrd-2.4.21-32.EL.img
Changing the default back to 0 has no effect, it still boots the 2.4.21-32.0.1.EL kernel and not the required SMP one. However, if we use the interactive GRUB boot menu & select the correct kernel interactively, it then boots SMP OK with both processors and all memory available.
I tried the obvious ploy of removing the last three kernel entries in grub.conf & setting default=0 but it still manages to boot the 2.4.21-32.0.1.EL UP kernel even though it is no longer in the kernel menu list.
We think we will disable automatic yum kernel updates in future , but meanwhile, has anyone any suggestions or experiences to share on this apart from a complete re-install of each affected node?
Les Oswald