Thank you very much for your replies and suggestions!
Turns out I have a broken RAID. I checked the failed out
drive by mounting read-only the /boot partition, and it
is configured to boot the older kernel version (the one the
system actually boots).
Like Phil said, the OS is seeing one thing, and GRUB another.
Questions:
1. How do I fix the array? (How do I put the failed out
drive back in? (I hope it is a small failure that the
software RAID can recover from, like a few bad blocks or
something. Otherwise I am willing to replace the drive.)
# mdadm /dev/md0 --add /dev/sda1
mdadm: Cannot open /dev/sda1: Device or resource busy
#
Maybe it's busy because the system really booted off it?
Maybe I can edit grub.conf to change hd(0,0) to hd (1,0)
and reboot. Where do I do that, in /dev/sda1 or /dev/sdb1?
I guess I could do it in both places. What do you think?
Note: I was able to add /dev/sda3 to /dev/md1, and it is
resync'ing the array now.
# mdadm /dev/md1 --add /dev/sda3
mdadm: re-added /dev/sda3
#
2. Is there a different configuration I should adopt, so that
OS and GRUB agree on what device to boot from? Or is this the
price I pay for using software RAID rather than HW RAID?
Data:
The /etc/grub.conf sym link is set up correctly:
lrwxrwxrwx 1 root root 22 Mar 17 2009 /etc/grub.conf ->
../boot/grub/grub.conf
My /boot filesystem lives on a RAID 1 array:
/dev/md0 on /boot type ext3 (rw)
/proc/mdstat shows only /dev/sdb is still in the RAID 1 mirror:
Personalities : [raid1]
md0 : active raid1 sdb1[1]
104320 blocks [2/1] [_U]
md1 : active raid1 sdb3[1]
275964480 blocks [2/1] [_U]
unused devices: <none>
For some reason, it does not show "F" for disk failure. I did
reboot the system a couple of times, maybe it forgot about the
failure. Older logwatch reports do have the F on both arrays.
lshw and "fdisk -l" shows both /dev/sda and /dev/sdb.
so does lsscsi:
[1:0:0:0] disk IBM-ESXS ST3300007LC FN B26B /dev/sda
[1:0:1:0] disk IBM-ESXS MAT3300NC FN B414 /dev/sdb
[1:0:8:0] process IBM 39M6750a S320 0 1 -
Thanks very much for the help!
Best,
Aleksey