[CentOS] (C8) root on mdraid

Sun Nov 15 11:32:45 UTC 2020
Łukasz Posadowski <mail at lukaszposadowski.pl>

Hello everyone. 

I'm trying to install CentOS 8 with root and swap partitions on
software raid. The plan is:
- create md0 raid level 1 with 2 hard drives: /dev/sda and /dev/sdb,
using Linux Rscue CD,
- install CentOS 8 with Virtual Box on my laptop,
- rsync CentOS 8 root partition on /dev/md0p1,
- chroot in CentOS 8 root partition,
- configure /etc/mdadm.conf, grub.cfg, initramfs, install bootloader on
both sda and sdb drives.

I think I can do first four of the above, but my CentOS installation
acts strange after rebooting the server. It recognizes the raid, but
boots randomly with root on /dev/sda1 (and recognizes raid
with /dev/sdb disk), or with root on /dev/sdb1 (and recognizes raid
with /dev/sda disk). When booting from Linux Rescue CD, the raid with
two disk is recognized.



Creating md0 wth Linux Rescue CD.

# lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0   7:0    0 819.5M  1 loop /run/archiso/sfs/airootfs
sda     8:0    0 232.9G  0 disk 
sdb     8:16   0 232.9G  0 disk 
sr0    11:0    1   871M  0 rom  

# mdadm --create /dev/md0 --level=1 --raid-devices=2 
--metadata=0.9 /dev/sda /dev/sdb
mdadm: array /dev/md0 started.

# mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=fef2f68b:
1a462c80:c44c77eb:7ee19756 devices=/dev/sda,/dev/sdb

# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb[1] sda[0]
      244198464 blocks [2/2] [UU]
      [>....................]  resync =  1.9% (4682944/244198464)
finish=52.2min speed=76356K/sec bitmap: 2/2 pages [8KB], 65536KB chunk

unused devices: <none>

(1 hour later)

# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb[1] sda[0]
      244198464 blocks [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices: <none>

# lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0   7:0    0 819.5M  1 loop  /run/archiso/sfs/airootfs
sda     8:0    0 232.9G  0 disk  
└─md0   9:0    0 232.9G  0 raid1 
sdb     8:16   0 232.9G  0 disk  
└─md0   9:0    0 232.9G  0 raid1 
sr0    11:0    1   871M  0 rom  

---------------------------------

So far so good. Now partitions:

# fdisk /dev/md0

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x64998720.

Command (m for help): o
Created a new DOS disklabel with disk identifier 0xfe784f57.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-488396927, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-488396927, default
488396927): +220G

Created a new partition 1 of type 'Linux' and of size 220 GiB.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 
First sector (461375488-488396927, default 461375488): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (461375488-488396927,
default 488396927): +4G

Created a new partition 2 of type 'Linux' and of size 4 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.


# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0       7:0    0 819.5M  1 loop  /run/archiso/sfs/airootfs
sda         8:0    0 232.9G  0 disk  
└─md0       9:0    0 232.9G  0 raid1 
  ├─md0p1 259:0    0   220G  0 part  
  └─md0p2 259:1    0     4G  0 part  
sdb         8:16   0 232.9G  0 disk  
└─md0       9:0    0 232.9G  0 raid1 
  ├─md0p1 259:0    0   220G  0 part  
  └─md0p2 259:1    0     4G  0 part  
sr0        11:0    1   871M  0 rom   


# mkfs -t ext4 /dev/md0p1
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 57671680 4k blocks and 14417920 inodes
Filesystem UUID: 511d0d07-4d55-4b6c-939e-d23ae2da1ce0
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208, 4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done     

# mkswap /dev/md0p2 
Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
no label, UUID=5667d007-0d3e-4b54-8a72-6ade52609adc

--------------------------------------

Mount would be root partition:

# mkdir /mnt/centos8
# mount -t ext4 /dev/md0p1 /mnt/centos8/

--------------------------------------

OK, I have what I need. Now log into virtualised CentOS 8 in Virtual
Box.

# rpm -q rsync
rsync-3.1.3-7.el8.x86_64
# rpm -q mdadm
mdadm-4.1-13.el8.x86_64

# rsync -vaxAXHSz --delete / 192.168.111.167:/mnt/centos8/
The authenticity of host '192.168.111.167 (192.168.111.167)' can't be
established. ECDSA key fingerprint is
SHA256:HukswAQlarthYqbpVHIHm80JaWHKX12AxgOn8UuKQxk. Are you sure you
want to continue connecting (yes/no/[fingerprint])? yes Warning:
Permanently added '192.168.111.167' (ECDSA) to the list of known hosts.
root at 192.168.111.167's password: sending incremental file list
./
bin -> usr/bin
lib -> usr/lib
lib64 -> usr/lib64
sbin -> usr/sbin
[lots and lots of files...]
var/yp/

sent 694,980,592 bytes  received 745,802 bytes  1,795,422.95 bytes/sec
total size is 1,558,934,091  speedup is 2.24

--------------------------------------

Chrooting time:

# mount --types proc /proc /mnt/centos8/proc
# mount --rbind /sys /mnt/centos8/sys
# mount --make-rslave /mnt/centos8/sys
# mount --rbind /dev /mnt/centos8/dev
# mount --make-rslave /mnt/centos8/dev
# chroot /mnt/centos8 /usr/bin/bash
# source /etc/profile


Configure raid: 

# mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=fef2f68b:
1a462c80:c44c77eb:7ee19756 devices=/dev/sda,/dev/sdb
# mdadm --detail --scan --verbose > /etc/mdadm.conf


Edit /etc/fstab:

# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Sun Nov  8 10:05:31 2020
#
# Accessible filesystems, by reference, are maintained under
# '/dev/disk/'. See man pages fstab(5), findfs(8), mount(8) and/or
# blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update
# systemd units generated from this file.
#
#UUID=36c5a887-79c3-4b6a-b433-0e4d7574b1e4 /
#ext4    defaults        1 1
/dev/md0p1 / ext4 defaults 1 1


Configure and install bootloader:

# cat /etc/default/grub 
GRUB_TIMEOUT=8
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

# grub2-mkconfig -o /boot/grub2/grub.cfg 
Generating grub configuration file ...
done

# grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
# grub2-install /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.


Configure initramfs:

# dracut -f /boot/initramfs-4.18.0-193.28.1.el8_2.x86_64.img
# 4.18.0-193.28.1.el8_2.x86_64

--------------------------------

Reboot and login to my new CentOS 8 installation:

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 232.9G  0 disk 
sdb      8:16   0 232.9G  0 disk 
├─sdb1   8:17   0   220G  0 part /
└─sdb2   8:18   0     4G  0 part 
sr0     11:0    1   871M  0 rom  

# mdadm --detail --scan --verbose
INACTIVE-ARRAY /dev/md127 num-devices=1 metadata=0.90 UUID=fef2f68b:
1a462c80:c44c77eb:7ee19756 devices=/dev/sda

# cat /proc/mdstat 
Personalities : 
md127 : inactive sda[0](S)
      244198464 blocks
       
unused devices: <none>


I can live with md127, because it happens constantly on multiple
distributions, but why root is on /dev/sdb1 (and sometimes
it's /dev/sda1)? I'm running out of spells now. :) Do anyone can
suggest what else I forgot to do?

thanks

-- 
Łukasz Posadowski