[CentOS] CentOS 6.2 software raid 10 with LVM - Need help with degraded drive and only one MBR

Sat Mar 3 06:57:52 UTC 2012
Devin Reade <gdr at gno.org>

Jonathan Vomacka <juvix88 at gmail.com> wrote:

> On 3/2/2012 3:09 PM, Devin Reade wrote:
>> Putting an MBR on all disks right after an OS install, as previously
>> mentioned, is of course the best option (although it's too late for that
>> in this instance).
> 
> In terms of the installing the MBR after the OS install, is this the responsibility of me, or the datacenter?

Whose *responsibility* it is is between you and your provider and
depends on your SLAs, etc.  It requires root access to perform.

> Also, would I need KVM access to do this or can it be done through SSH? My DC charges a rental for KVM.

You can do it as root via ssh while your server is fully running; it
does not need single user mode nor console access.  The following is an
extract from the changelog of one of my machines that has four disks,
each of which has a 200Mb partition 1 (for /boot, mirrored), with the
rest of each of the disks (partition 2) put into a four volume RAID6
set (which is used by LVM for the remaining filesystems).

        - made sure we have boot blocks on both disks, based on information
          at http://grub.enbug.org/MirroringRAID
                modified /boot/grub/device.map from:
                        (hd1)     /dev/sda
                        (hd0)     /dev/sdb
                        (hd2)     /dev/sdc
                        (hd3)     /dev/sdd
                to:
                        (hd0)     /dev/sda
                        (hd0)     /dev/sdb
                        (hd0)     /dev/sdc
                        (hd0)     /dev/sdd
                and then:
                        # grub
                        grub> device (hd0) /dev/sdb
                        grub> root (hd0,0)
                        grub> setup (hd0)
                        grub> quit
                then repeated it for sdc and sdd, and finally again for sda
                just to be paranoid

Keep in mind that those disks already have the OS installed on
them; writing the MBR like that doesn't affect the existing filesystems.

I use this kind of configuration in quite a few machines (sometimes
with only two disks using mirroring on both partitions of each disk)
and have verified that it works both in testing and the hard way 
... having lost my nominal boot disk due to errors and recovered 
by powering down (these aren't hot-swap disks), replacing the 
bad drive, booting in degraded mode, partitioning the new disk,
and then adding its partitions back into the applicable RAID sets.
Ignoring the actual sync time, that's less than 20 minutes work,
most of which is the disk swap.  (Make sure you know which disk
is the faulty one before you power down.  I always ensure that
there are markings on the drive that correlate to identifiers
used by Linux before putting drives into production; often the 
manufacturer's serial number or WWID is sufficient.)

>  "I'm sorry your having a hard time with software raid on your server and
>  our install process. From what I remember talking with out techs long
>  ago about this is, that when using raid10 and software raid, the
>  bootloader cannot be installed on the software raid partition and has to
>  be on a single drive.

As Robert already observed, that's so much bullshit unless the
hardware they're providing is so old (like probably > 5 years) that
the BIOS can't be told to boot from alternate disks, in sequence.
Even then, it's not insurmountable given they can change the boot
device or revert to flipping cables even on the oldest hardware.

Give that manager the instructions above and suggest that he add
it to his provisioning procedure.

Devin