[CentOS] *very* ugly mdadm issue

Tue Sep 2 18:33:00 UTC 2014
m.roth at 5-cent.us <m.roth at 5-cent.us>

I'm the OP, here....

Les Mikesell wrote:
> On Tue, Sep 2, 2014 at 12:42 PM, Warren Young <warren at etr-usa.com> wrote:
>> On 8/29/2014 14:26, m.roth at 5-cent.us wrote:
>>> Note that we did this on *raw*, unpartitioned drives (not my idea).
>>> I added /dev/sdc to /dev/md4, and it started rebuilding.
> I haven't used raw devices as members so I'm not sure I understand the
> scenario.   However, I thought that devices over 2TB would not auto
> assemble so you would have to manually add the ARRAY entry for
> /dev/md4 in /etc/mdadm.conf containing /dev/sdd  and /dev/sdc for the
> system to recognize it at bootup.
Yeah. That was one thing I discovered. Silly me, assuming that the mdadm
would create an entry in /etc/mdadm.conf. And this is not something I do
more than once or twice a year, and haven't this year (we have a good
number of Dells with a PERC 7, or then there's the JetStors....).

>> Are you certain /dev/sdc is partially overwritten now?  What happens if
>> you try to mount it?  If it mounts, go buy that second fresh disk, then
>> the mirror up correctly this time.

It was toast.
> But sdd _should_ have the correct data - it just isn't being detected
> as a raid member.   I think with smaller devices - or at least devices
> with smaller partitions and FD type in the MBR it would have worked
> automatically with the kernel autodetect.

Both had a GPT on them, just no partitions. And that's the thing that
really puzzles me - why mdadm couldn't find the RAID info on /dev/sdd,
which *had* been just fine.

Anyway, the upshot was my manager was rather annoyed - I *should* have
pulled sdc, and put in a new one, and just let that go. I still think it
would have failed, given the inability of mdadm to find the info on sdd.
We wound up just remaking the RAID, and rebuilding the mirror over the