Hello,
I have a RAID5 array on my CentOS 5.6 x86_64 workstation which "suddenly" failed to work (actually after the system could not resume from a suspend).
I had recently issues after moving the workstation to another office, where one of the disks got accidently unplugged. But the RAID was working and it had reconstructed (as far as I can tell) the data. After I replugged the disk, it was working normally over the last two days (enough to get back to a proper state I guess/hoped)
This RAID is used as an LVM volume group for all my important data, among them the root of the operating system(s). It based on four partitions on four separate disks (the third partition of each disk, 3 active, one spare)
When booting, I get an error message similar to:
raid5 failed: No md superblock detected on /dev/md0.
and the LVM volume group does not come up.
I then booted using the CentOS 5.6 LiveCD and tried to run a few mdadm command (see just below). It seems that there are some data still lying around, but I'm not very experienced with RAID and I thought that I would ask for advice before trying commands which may impact the data such as assemble, etc.
Thanks in advance for any help!
Mathieu
[root@livecd ~]# mdadm --misc -E /dev/md0 mdadm: No md superblock detected on /dev/md0. [root@livecd ~]# mdadm --misc -Q /dev/md0 /dev/md0: is an md device which is not active /dev/md0: No md super block found, not an md component. [root@livecd ~]# mdadm --misc -D /dev/md0 mdadm: md device /dev/md0 does not appear to be active. [root@livecd ~]# mdadm -E /dev/sd*3 /dev/sda3: Magic : a92b4efc Version : 0.90.00 UUID : 7533411a:f066a145:1e89d48e:1a8374a3 Creation Time : Tue Dec 1 12:01:05 2009 Raid Level : raid5 Used Dev Size : 204796416 (195.31 GiB 209.71 GB) Array Size : 409592832 (390.62 GiB 419.42 GB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 0
Update Time : Wed Aug 17 14:47:36 2011 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 1 Spare Devices : 0 Checksum : ed6d5dcd - correct Events : 38857
Layout : left-symmetric Chunk Size : 256K
Number Major Minor RaidDevice State this 0 8 3 0 active sync /dev/sda3
0 0 8 3 0 active sync /dev/sda3 1 1 0 0 1 faulty removed 2 2 8 51 2 active sync /dev/sdd3 /dev/sdb3: Magic : a92b4efc Version : 0.90.00 UUID : 7533411a:f066a145:1e89d48e:1a8374a3 Creation Time : Tue Dec 1 12:01:05 2009 Raid Level : raid5 Used Dev Size : 204796416 (195.31 GiB 209.71 GB) Array Size : 409592832 (390.62 GiB 419.42 GB) Raid Devices : 3 Total Devices : 4 Preferred Minor : 0
Update Time : Wed Aug 10 11:52:15 2011 State : clean Active Devices : 3 Working Devices : 4 Failed Devices : 0 Spare Devices : 1 Checksum : ed63a948 - correct Events : 9022
Layout : left-symmetric Chunk Size : 256K
Number Major Minor RaidDevice State this 1 8 19 1 active sync /dev/sdb3
0 0 8 3 0 active sync /dev/sda3 1 1 8 19 1 active sync /dev/sdb3 2 2 8 51 2 active sync /dev/sdd3 3 3 8 35 3 spare /dev/sdc3 /dev/sdc3: Magic : a92b4efc Version : 0.90.00 UUID : 7533411a:f066a145:1e89d48e:1a8374a3 Creation Time : Tue Dec 1 12:01:05 2009 Raid Level : raid5 Used Dev Size : 204796416 (195.31 GiB 209.71 GB) Array Size : 409592832 (390.62 GiB 419.42 GB) Raid Devices : 3 Total Devices : 3 Preferred Minor : 0
Update Time : Tue Aug 16 17:48:59 2011 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Checksum : ed6bf2b0 - correct Events : 10670
Layout : left-symmetric Chunk Size : 256K
Number Major Minor RaidDevice State this 1 8 35 1 active sync /dev/sdc3
0 0 8 3 0 active sync /dev/sda3 1 1 8 35 1 active sync /dev/sdc3 2 2 8 51 2 active sync /dev/sdd3 /dev/sdd3: Magic : a92b4efc Version : 0.90.00 UUID : 7533411a:f066a145:1e89d48e:1a8374a3 Creation Time : Tue Dec 1 12:01:05 2009 Raid Level : raid5 Used Dev Size : 204796416 (195.31 GiB 209.71 GB) Array Size : 409592832 (390.62 GiB 419.42 GB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 0
Update Time : Wed Aug 17 14:47:36 2011 State : active Active Devices : 2 Working Devices : 2 Failed Devices : 1 Spare Devices : 0 Checksum : ed6d5e01 - correct Events : 38857
Layout : left-symmetric Chunk Size : 256K
Number Major Minor RaidDevice State this 2 8 51 2 active sync /dev/sdd3
0 0 8 3 0 active sync /dev/sda3 1 1 0 0 1 faulty removed 2 2 8 51 2 active sync /dev/sdd3 [root@livecd ~]#
[root@livecd ~]# mdadm --misc -E /dev/md0 mdadm: No md superblock detected on /dev/md0. [root@livecd ~]# mdadm --misc -Q /dev/md0 /dev/md0: is an md device which is not active /dev/md0: No md super block found, not an md component. [root@livecd ~]# mdadm --misc -D /dev/md0 mdadm: md device /dev/md0 does not appear to be active.
I could fix the issue.
Since the information on internet is a bit messy and scary here is a summary of the problem and of the solution, for future reference:
## PROBLEM Due to some failure related to suspend the RAID5 array became inconsistent. The symptom was that the superblock (which allow auto-configuration of the RAID array) was not recognized.
But the underlying member partitions were still recognized as per the following command:
[root@livecd ~]# mdadm -E /dev/sd*3
which provided details about their states
## SOLUTION (detailed output of the commands at the end of the post)
# Assemble the array with force option mdadm -v --assemble --force /dev/md0 /dev/sd{a,b,c,d}3
# NOTE: assembling without force option was not enough [root@livecd ~]# mdadm -v --assemble /dev/md0 /dev/sd{a,b,c,d}3 ... mdadm: /dev/md0 assembled from 2 drives - not enough to start the array while not clean - consider --force.
# Check state mdadm -D /dev/md0 ... Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 0 0 1 removed 2 8 51 2 active sync /dev/sdd3
# Add missing partitions to the array mdadm /dev/md0 -a /dev/sdb3 mdadm /dev/md0 -a /dev/sdc3
# Check that it is now OK mdadm -D /dev/md0 ... Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 8 19 1 spare rebuilding /dev/sdb3 2 8 51 2 active sync /dev/sdd3
3 8 35 - spare /dev/sdc3
# and watch it rebuild: ... Rebuild Status : 3% complete
Many thanks to the CentOS LiveCD team, they saved the day.
Cheers,
Mathieu
## DETAILED OUTPUT
[root@livecd ~]# mdadm -v --assemble --force /dev/md0 /dev/sd{a,b,c,d}3 mdadm: looking for devices for /dev/md0 mdadm: /dev/sda3 is identified as a member of /dev/md0, slot 0. mdadm: /dev/sdb3 is identified as a member of /dev/md0, slot 1. mdadm: /dev/sdc3 is identified as a member of /dev/md0, slot 1. mdadm: /dev/sdd3 is identified as a member of /dev/md0, slot 2. mdadm: added /dev/sdc3 to /dev/md0 as 1 mdadm: added /dev/sdd3 to /dev/md0 as 2 mdadm: added /dev/sda3 to /dev/md0 as 0 mdadm: /dev/md0 has been started with 2 drives (out of 3).
[root@livecd ~]# mdadm -D /dev/md0 /dev/md0: Version : 0.90 Creation Time : Tue Dec 1 12:01:05 2009 Raid Level : raid5 Array Size : 409592832 (390.62 GiB 419.42 GB) Used Dev Size : 204796416 (195.31 GiB 209.71 GB) Raid Devices : 3 Total Devices : 2 Preferred Minor : 0 Persistence : Superblock is persistent
Update Time : Wed Aug 17 14:47:36 2011 State : clean, degraded Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0
Layout : left-symmetric Chunk Size : 256K
UUID : 7533411a:f066a145:1e89d48e:1a8374a3 Events : 0.38857
Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 0 0 1 removed 2 8 51 2 active sync /dev/sdd3
[root@livecd ~]# mdadm /dev/md0 -a /dev/sdb3 mdadm: re-added /dev/sdb3 [root@livecd ~]# mdadm /dev/md0 -a /dev/sdc3 mdadm: added /dev/sdc3 [root@livecd ~]# mdadm -D /dev/md0 /dev/md0: Version : 0.90 Creation Time : Tue Dec 1 12:01:05 2009 Raid Level : raid5 Array Size : 409592832 (390.62 GiB 419.42 GB) Used Dev Size : 204796416 (195.31 GiB 209.71 GB) Raid Devices : 3 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent
Update Time : Wed Aug 17 14:47:36 2011 State : clean, degraded, recovering Active Devices : 2 Working Devices : 4 Failed Devices : 0 Spare Devices : 2
Layout : left-symmetric Chunk Size : 256K
Rebuild Status : 0% complete
UUID : 7533411a:f066a145:1e89d48e:1a8374a3 Events : 0.38857
Number Major Minor RaidDevice State 0 8 3 0 active sync /dev/sda3 1 8 19 1 spare rebuilding /dev/sdb3 2 8 51 2 active sync /dev/sdd3
3 8 35 - spare /dev/sdc3
-
Mathieu Baudier