[CentOS] *very* ugly mdadm issue

On 8/29/2014 14:26, m.roth at 5-cent.us wrote:
>
> Note that we did this on *raw*, unpartitioned drives (not my idea).

Nothing wrong with that, particularly with big "midden" volumes like 
this one.

> I added
> /dev/sdc to /dev/md4, and it started rebuilding.

*facepalm*

You forgot the primary maxim of data integrity: two is one, one is none.

When you overwrote your original copy with what you thought was a clone, 
you reduced yourself to a single copy again.  If anything is wrong with 
that copy, you now have two copies of the error.

What you *should* have done is buy two drives, set them up as a new 
mirror, copy the data over to them, then pull the old /dev/sdc and put 
it on a shelf as an offline archive mirror.  /dev/sdc has probably 
already given you its rated service life, so it's not like you're really 
saving money here.  The drive has already depreciated to zero.

You're probably going to spend more in terms of your time (salary + 
benefits) to fix this than the extra drive would have cost you, and at 
the end of it, you still won't have the security of that offline archive 
mirror.

I know this isn't the answer you wanted, but it's probably the answer a 
lot of people *wanted* to give, but chose not to, going by the crickets. 
  (It's either that or the 3-day holiday weekend.)

I don't know how much I can help you.  I have always used hardware RAID 
on Linux, even for simple mirrors.

I don't see why it matters that your /dev/sdd partitioning is different 
from your /dev/sdc.  When you told it to blast /dev/sdc with the 
contents of /dev/sdd, it should have copied the partitioning, too.

Are you certain /dev/sdc is partially overwritten now?  What happens if 
you try to mount it?  If it mounts, go buy that second fresh disk, then 
set the mirror up correctly this time.