[CentOS] mdraid woes (missing superblock?)

Tue Apr 26 14:18:04 UTC 2011
Jussi Hirvi <listmember at greenspot.fi>

I have a raid1 array which is somehow faulty. There is 1,5 TB of stuff, 
I would not want to lose it (though I have full backup).

The array cannot be mounted on startup (error message was "missing 
superblock"). I had to boot from DVD with linux rescue and remove the 
array from fstab.

Here is some info - I am a little dumbfounded.

[root at a134-224 log]# cat /proc/mdstat
(...)
md5 : active raid1 sdb1[0] sdd1[1]
       1562505856 blocks [2/2] [UU]


## mdadm examine - I believe the following output indicates that there 
really *is* a superblock

[root at a134-224 log]# mdadm --examine /dev/sdb1

/dev/sdb1:
           Magic : a92b4efc
         Version : 0.90.00
            UUID : 194e7fdc:2a288e87:c03d8b24:c0120803
   Creation Time : Sat Apr 23 20:41:08 2011
      Raid Level : raid1
   Used Dev Size : 1562505856 (1490.12 GiB 1600.01 GB)
      Array Size : 1562505856 (1490.12 GiB 1600.01 GB)
    Raid Devices : 2
   Total Devices : 2
Preferred Minor : 5

     Update Time : Tue Apr 26 11:48:35 2011
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0
        Checksum : 657d7a24 - correct
          Events : 14


       Number   Major   Minor   RaidDevice State
this     0       8       17        0      active sync   /dev/sdb1

    0     0       8       17        0      active sync   /dev/sdb1
    1     1       8       49        1      active sync   /dev/sdd1


# examine the other partition/dev/sdd1:
           Magic : a92b4efc
         Version : 0.90.00
            UUID : 194e7fdc:2a288e87:c03d8b24:c0120803
   Creation Time : Sat Apr 23 20:41:08 2011
      Raid Level : raid1
   Used Dev Size : 1562505856 (1490.12 GiB 1600.01 GB)
      Array Size : 1562505856 (1490.12 GiB 1600.01 GB)
    Raid Devices : 2
   Total Devices : 2
Preferred Minor : 5

     Update Time : Tue Apr 26 11:48:35 2011
           State : clean
  Active Devices : 2
Working Devices : 2
  Failed Devices : 0
   Spare Devices : 0
        Checksum : 657d7a46 - correct
          Events : 14


       Number   Major   Minor   RaidDevice State
this     1       8       49        1      active sync   /dev/sdd1

    0     0       8       17        0      active sync   /dev/sdb1
    1     1       8       49        1      active sync   /dev/sdd1



[root at a134-224 log]# mdadm --examine --scan
(...)
ARRAY /dev/md5 level=raid1 num-devices=2 
UUID=194e7fdc:2a288e87:c03d8b24:c0120803


[root at a134-224 log]# mdadm --stop /dev/md5
mdadm: stopped /dev/md5

[root at a134-224 log]# mdadm --assemble /dev/md5
mdadm: /dev/md5 has been started with 2 drives.

# but it cannot be mounted:

[root at a134-224 log]# mount /dev/md5 /kvbck/
mount: you must specify the filesystem type
[root at a134-224 log]# mount -t ext3 /dev/md5 /kvbck/
mount: wrong fs type, bad option, bad superblock on /dev/md5,
        missing codepage or other error
        In some cases useful info is found in syslog - try
        dmesg | tail  or so

[root at a134-224 log]# tail /var/log/messages
Apr 26 17:14:09 localhost kernel: md: unbind<sdb1>
Apr 26 17:14:09 localhost kernel: md: export_rdev(sdb1)
Apr 26 17:14:09 localhost kernel: md: unbind<sdd1>
Apr 26 17:14:09 localhost kernel: md: export_rdev(sdd1)
Apr 26 17:14:17 localhost kernel: md: md5 stopped.
Apr 26 17:14:17 localhost kernel: md: bind<sdd1>
Apr 26 17:14:17 localhost kernel: md: bind<sdb1>
Apr 26 17:14:17 localhost kernel: raid1: raid set md5 active with 2 out 
of 2 mirrors
Apr 26 17:14:56 localhost kernel: hfs: unable to find HFS+ superblock
Apr 26 17:15:05 localhost kernel: VFS: Can't find ext3 filesystem on dev 
md5.

# Why does the log complain about HFS+ superblock? The fs should be ext3.

- Jussi

-- 
Jussi Hirvi * Green Spot
Suvilahdenkatu 1 B 78 * 00500 Helsinki * Finland
Tel. +358 9 493 981 * Mobile +358 40 771 2098 (only sms)
jussi.hirvi at greenspot.fi * http://www.greenspot.fi