[CentOS] *very* ugly mdadm issue

[m.roth at 5-cent.us]

I'm the OP, here....

Les Mikesell wrote:
> On Tue, Sep 2, 2014 at 12:42 PM, Warren Young <warren at etr-usa.com> wrote:
>> On 8/29/2014 14:26, m.roth at 5-cent.us wrote:
>>>
>>> Note that we did this on *raw*, unpartitioned drives (not my idea).
>>
>>> I added /dev/sdc to /dev/md4, and it started rebuilding.
<snip>
> I haven't used raw devices as members so I'm not sure I understand the
> scenario.   However, I thought that devices over 2TB would not auto
> assemble so you would have to manually add the ARRAY entry for
> /dev/md4 in /etc/mdadm.conf containing /dev/sdd  and /dev/sdc for the
> system to recognize it at bootup.
>
Yeah. That was one thing I discovered. Silly me, assuming that the mdadm
would create an entry in /etc/mdadm.conf. And this is not something I do
more than once or twice a year, and haven't this year (we have a good
number of Dells with a PERC 7, or then there's the JetStors....).

>> Are you certain /dev/sdc is partially overwritten now?  What happens if
>> you try to mount it?  If it mounts, go buy that second fresh disk, then
set
>> the mirror up correctly this time.

It was toast.
>
> But sdd _should_ have the correct data - it just isn't being detected
> as a raid member.   I think with smaller devices - or at least devices
> with smaller partitions and FD type in the MBR it would have worked
> automatically with the kernel autodetect.

Both had a GPT on them, just no partitions. And that's the thing that
really puzzles me - why mdadm couldn't find the RAID info on /dev/sdd,
which *had* been just fine.

Anyway, the upshot was my manager was rather annoyed - I *should* have
pulled sdc, and put in a new one, and just let that go. I still think it
would have failed, given the inability of mdadm to find the info on sdd.
We wound up just remaking the RAID, and rebuilding the mirror over the
weekend.

      mark