Hello everyone.
I'm trying to install CentOS 8 with root and swap partitions on software raid. The plan is: - create md0 raid level 1 with 2 hard drives: /dev/sda and /dev/sdb, using Linux Rscue CD, - install CentOS 8 with Virtual Box on my laptop, - rsync CentOS 8 root partition on /dev/md0p1, - chroot in CentOS 8 root partition, - configure /etc/mdadm.conf, grub.cfg, initramfs, install bootloader on both sda and sdb drives.
I think I can do first four of the above, but my CentOS installation acts strange after rebooting the server. It recognizes the raid, but boots randomly with root on /dev/sda1 (and recognizes raid with /dev/sdb disk), or with root on /dev/sdb1 (and recognizes raid with /dev/sda disk). When booting from Linux Rescue CD, the raid with two disk is recognized.
Creating md0 wth Linux Rescue CD.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 819.5M 1 loop /run/archiso/sfs/airootfs sda 8:0 0 232.9G 0 disk sdb 8:16 0 232.9G 0 disk sr0 11:0 1 871M 0 rom
# mdadm --create /dev/md0 --level=1 --raid-devices=2 --metadata=0.9 /dev/sda /dev/sdb mdadm: array /dev/md0 started.
# mdadm --detail --scan --verbose ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=fef2f68b: 1a462c80:c44c77eb:7ee19756 devices=/dev/sda,/dev/sdb
# cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb[1] sda[0] 244198464 blocks [2/2] [UU] [>....................] resync = 1.9% (4682944/244198464) finish=52.2min speed=76356K/sec bitmap: 2/2 pages [8KB], 65536KB chunk
unused devices: <none>
(1 hour later)
# cat /proc/mdstat Personalities : [raid1] md0 : active raid1 sdb[1] sda[0] 244198464 blocks [2/2] [UU] bitmap: 0/2 pages [0KB], 65536KB chunk
unused devices: <none>
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 819.5M 1 loop /run/archiso/sfs/airootfs sda 8:0 0 232.9G 0 disk └─md0 9:0 0 232.9G 0 raid1 sdb 8:16 0 232.9G 0 disk └─md0 9:0 0 232.9G 0 raid1 sr0 11:0 1 871M 0 rom
---------------------------------
So far so good. Now partitions:
# fdisk /dev/md0
Welcome to fdisk (util-linux 2.33.1). Changes will remain in memory only, until you decide to write them. Be careful before using the write command.
Device does not contain a recognized partition table. Created a new DOS disklabel with disk identifier 0x64998720.
Command (m for help): o Created a new DOS disklabel with disk identifier 0xfe784f57.
Command (m for help): n Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): p Partition number (1-4, default 1): First sector (2048-488396927, default 2048): Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-488396927, default 488396927): +220G
Created a new partition 1 of type 'Linux' and of size 220 GiB.
Command (m for help): n Partition type p primary (1 primary, 0 extended, 3 free) e extended (container for logical partitions) Select (default p): p Partition number (2-4, default 2): First sector (461375488-488396927, default 461375488): Last sector, +/-sectors or +/-size{K,M,G,T,P} (461375488-488396927, default 488396927): +4G
Created a new partition 2 of type 'Linux' and of size 4 GiB.
Command (m for help): w The partition table has been altered. Calling ioctl() to re-read partition table. Syncing disks.
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT loop0 7:0 0 819.5M 1 loop /run/archiso/sfs/airootfs sda 8:0 0 232.9G 0 disk └─md0 9:0 0 232.9G 0 raid1 ├─md0p1 259:0 0 220G 0 part └─md0p2 259:1 0 4G 0 part sdb 8:16 0 232.9G 0 disk └─md0 9:0 0 232.9G 0 raid1 ├─md0p1 259:0 0 220G 0 part └─md0p2 259:1 0 4G 0 part sr0 11:0 1 871M 0 rom
# mkfs -t ext4 /dev/md0p1 mke2fs 1.44.5 (15-Dec-2018) Creating filesystem with 57671680 4k blocks and 14417920 inodes Filesystem UUID: 511d0d07-4d55-4b6c-939e-d23ae2da1ce0 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000, 7962624, 11239424, 20480000, 23887872
Allocating group tables: done Writing inode tables: done Creating journal (262144 blocks): done Writing superblocks and filesystem accounting information: done
# mkswap /dev/md0p2 Setting up swapspace version 1, size = 4 GiB (4294963200 bytes) no label, UUID=5667d007-0d3e-4b54-8a72-6ade52609adc
--------------------------------------
Mount would be root partition:
# mkdir /mnt/centos8 # mount -t ext4 /dev/md0p1 /mnt/centos8/
--------------------------------------
OK, I have what I need. Now log into virtualised CentOS 8 in Virtual Box.
# rpm -q rsync rsync-3.1.3-7.el8.x86_64 # rpm -q mdadm mdadm-4.1-13.el8.x86_64
# rsync -vaxAXHSz --delete / 192.168.111.167:/mnt/centos8/ The authenticity of host '192.168.111.167 (192.168.111.167)' can't be established. ECDSA key fingerprint is SHA256:HukswAQlarthYqbpVHIHm80JaWHKX12AxgOn8UuKQxk. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.111.167' (ECDSA) to the list of known hosts. root@192.168.111.167's password: sending incremental file list ./ bin -> usr/bin lib -> usr/lib lib64 -> usr/lib64 sbin -> usr/sbin [lots and lots of files...] var/yp/
sent 694,980,592 bytes received 745,802 bytes 1,795,422.95 bytes/sec total size is 1,558,934,091 speedup is 2.24
--------------------------------------
Chrooting time:
# mount --types proc /proc /mnt/centos8/proc # mount --rbind /sys /mnt/centos8/sys # mount --make-rslave /mnt/centos8/sys # mount --rbind /dev /mnt/centos8/dev # mount --make-rslave /mnt/centos8/dev # chroot /mnt/centos8 /usr/bin/bash # source /etc/profile
Configure raid:
# mdadm --detail --scan --verbose ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=fef2f68b: 1a462c80:c44c77eb:7ee19756 devices=/dev/sda,/dev/sdb # mdadm --detail --scan --verbose > /etc/mdadm.conf
Edit /etc/fstab:
# cat /etc/fstab
# # /etc/fstab # Created by anaconda on Sun Nov 8 10:05:31 2020 # # Accessible filesystems, by reference, are maintained under # '/dev/disk/'. See man pages fstab(5), findfs(8), mount(8) and/or # blkid(8) for more info. # # After editing this file, run 'systemctl daemon-reload' to update # systemd units generated from this file. # #UUID=36c5a887-79c3-4b6a-b433-0e4d7574b1e4 / #ext4 defaults 1 1 /dev/md0p1 / ext4 defaults 1 1
Configure and install bootloader:
# cat /etc/default/grub GRUB_TIMEOUT=8 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="rhgb quiet" GRUB_DISABLE_RECOVERY="true" GRUB_ENABLE_BLSCFG=true
# grub2-mkconfig -o /boot/grub2/grub.cfg Generating grub configuration file ... done
# grub2-install /dev/sda Installing for i386-pc platform. Installation finished. No error reported. # grub2-install /dev/sdb Installing for i386-pc platform. Installation finished. No error reported.
Configure initramfs:
# dracut -f /boot/initramfs-4.18.0-193.28.1.el8_2.x86_64.img # 4.18.0-193.28.1.el8_2.x86_64
--------------------------------
Reboot and login to my new CentOS 8 installation:
# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 232.9G 0 disk sdb 8:16 0 232.9G 0 disk ├─sdb1 8:17 0 220G 0 part / └─sdb2 8:18 0 4G 0 part sr0 11:0 1 871M 0 rom
# mdadm --detail --scan --verbose INACTIVE-ARRAY /dev/md127 num-devices=1 metadata=0.90 UUID=fef2f68b: 1a462c80:c44c77eb:7ee19756 devices=/dev/sda
# cat /proc/mdstat Personalities : md127 : inactive sda[0](S) 244198464 blocks
unused devices: <none>
I can live with md127, because it happens constantly on multiple distributions, but why root is on /dev/sdb1 (and sometimes it's /dev/sda1)? I'm running out of spells now. :) Do anyone can suggest what else I forgot to do?
thanks
On 11/15/20 3:32 AM, Łukasz Posadowski wrote:
Do anyone can suggest what else I forgot to do?
Use metadata version 1.2 instead of 0.9.
You need for the filesystem to be not visible until after the RAID is assembled, and the easiest way to do that is to put the metadata at the beginning of the drive and the partition table inside the RAID volume.
With metadata version 0.9, the partition table is technically inside the volume, but it's at the same offset that it would be for a disk with no RAID volume, so it can be recognized before the volume is assembled.
Sun, 15 Nov 2020 14:16:48 -0800 Gordon Messmer gordon.messmer@gmail.com:
On 11/15/20 3:32 AM, Łukasz Posadowski wrote:
Do anyone can suggest what else I forgot to do?
Use metadata version 1.2 instead of 0.9.
You need for the filesystem to be not visible until after the RAID is assembled, and the easiest way to do that is to put the metadata at the beginning of the drive and the partition table inside the RAID volume.
With metadata version 0.9, the partition table is technically inside the volume, but it's at the same offset that it would be for a disk with no RAID volume, so it can be recognized before the volume is assembled.
Thanks, I'll try that. I'm use to metadata 0.9, because GRUB have (had?) some issue with the newer ones.
Hi,
I am curious, what config your /etc/grub2.cfg looks like?
does it have the correct root partition specified under "set root=" or in linux option or how it looks in boot loader during a start?
Thanks
On Mon, 16 Nov 2020 at 08:41, Łukasz Posadowski mail@lukaszposadowski.pl wrote:
Sun, 15 Nov 2020 14:16:48 -0800 Gordon Messmer gordon.messmer@gmail.com:
On 11/15/20 3:32 AM, Łukasz Posadowski wrote:
Do anyone can suggest what else I forgot to do?
Use metadata version 1.2 instead of 0.9.
You need for the filesystem to be not visible until after the RAID is assembled, and the easiest way to do that is to put the metadata at the beginning of the drive and the partition table inside the RAID volume.
With metadata version 0.9, the partition table is technically inside the volume, but it's at the same offset that it would be for a disk with no RAID volume, so it can be recognized before the volume is assembled.
Thanks, I'll try that. I'm use to metadata 0.9, because GRUB have (had?) some issue with the newer ones.
-- Łukasz Posadowski _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 11/15/20 10:40 PM, Łukasz Posadowski wrote:
Sun, 15 Nov 2020 14:16:48 -0800 Gordon Messmer gordon.messmer@gmail.com:
Use metadata version 1.2 instead of 0.9.
Thanks, I'll try that. I'm use to metadata 0.9, because GRUB have (had?) some issue with the newer ones.
If that doesn't work, and you need to use metadata 0.9, then check /etc/default/grub and make sure that GRUB_CMDLINE_LINUX contains "rd.md.uuid=<THE UUID OF YOUR md0>".
Mon, 16 Nov 2020 14:28:54 -0800 Gordon Messmer gordon.messmer@gmail.com:
On 11/15/20 10:40 PM, Łukasz Posadowski wrote:
Sun, 15 Nov 2020 14:16:48 -0800 Gordon Messmer gordon.messmer@gmail.com:
Use metadata version 1.2 instead of 0.9.
Thanks, I'll try that. I'm use to metadata 0.9, because GRUB have (had?) some issue with the newer ones.
If that doesn't work, and you need to use metadata 0.9, then check /etc/default/grub and make sure that GRUB_CMDLINE_LINUX contains "rd.md.uuid=<THE UUID OF YOUR md0>".
T.H.A.N.K Y.O.U. :) I was sure I had to pass uuid of the partition, not md0. It is working now. One last thing: since Linux Rescue CD does not recognize selinux, / mounted itself as read-only. Disabling SeLinux helped. I think I need to relabel all the files on a new host. So generally: - edit /etc/default/grub and grub2-mkconfig, - generate mdadm.conf (I think a copy of it resides inside initramfs), - fstab (root mounts anyway, so just for swap patition), - disable selinux, - generate initramfs with dracut.
Big thanks to all of you. It was rough, but it's done.
In article 20201115123245.db62b8248e1f248afe02844a@lukaszposadowski.pl, Lukasz Posadowski mail@lukaszposadowski.pl wrote:
Hello everyone.
I'm trying to install CentOS 8 with root and swap partitions on software raid. The plan is:
- create md0 raid level 1 with 2 hard drives: /dev/sda and /dev/sdb,
using Linux Rscue CD,
- install CentOS 8 with Virtual Box on my laptop,
- rsync CentOS 8 root partition on /dev/md0p1,
- chroot in CentOS 8 root partition,
- configure /etc/mdadm.conf, grub.cfg, initramfs, install bootloader on
both sda and sdb drives.
I think I can do first four of the above, but my CentOS installation acts strange after rebooting the server. It recognizes the raid, but boots randomly with root on /dev/sda1 (and recognizes raid with /dev/sdb disk), or with root on /dev/sdb1 (and recognizes raid with /dev/sda disk). When booting from Linux Rescue CD, the raid with two disk is recognized.
I thought it was much more usual to partition both disks to give sda1,2,3 and sdb1,2,3, and then create /dev/md0 from sda1/sdb1, /dev/md1 from sda2/sdb3, and so on.
That's the way I have always done it, and have never had any problems. Never seen an attempt to partition an md device before. In that case, how would the kernel and initrd be found in order to assemble the RAID?
Cheers Tony
On Mon, Nov 16, 2020, 2:29 AM Tony Mountifield tony@softins.co.uk wrote:
I thought it was much more usual to partition both disks to give sda1,2,3 and sdb1,2,3, and then create /dev/md0 from sda1/sdb1, /dev/md1 from sda2/sdb3, and so on.
What I always did was to mdraid a single full disk partition then use lvm for any file systems. Boot disks did need a separate /boot partition.
Mon, 16 Nov 2020 10:29:17 +0000 Tony Mountifield tony@softins.co.uk:
In article 20201115123245.db62b8248e1f248afe02844a@lukaszposadowski.pl, Lukasz Posadowski mail@lukaszposadowski.pl wrote:
Hello everyone.
I'm trying to install CentOS 8 with root and swap partitions on software raid. The plan is:
- create md0 raid level 1 with 2 hard drives: /dev/sda and /dev/sdb,
using Linux Rscue CD,
- install CentOS 8 with Virtual Box on my laptop,
- rsync CentOS 8 root partition on /dev/md0p1,
- chroot in CentOS 8 root partition,
- configure /etc/mdadm.conf, grub.cfg, initramfs, install
bootloader on both sda and sdb drives.
I think I can do first four of the above, but my CentOS installation acts strange after rebooting the server. It recognizes the raid, but boots randomly with root on /dev/sda1 (and recognizes raid with /dev/sdb disk), or with root on /dev/sdb1 (and recognizes raid with /dev/sda disk). When booting from Linux Rescue CD, the raid with two disk is recognized.
I thought it was much more usual to partition both disks to give sda1,2,3 and sdb1,2,3, and then create /dev/md0 from sda1/sdb1, /dev/md1 from sda2/sdb3, and so on.
That's the way I have always done it, and have never had any problems. Never seen an attempt to partition an md device before. In that case, how would the kernel and initrd be found in order to assemble the RAID?
It totaly works either way - raid on partitions and raid on disks. I'm use to mirroring whole disks.
-
Gordon Messmer -
John Pierce -
Ruslanas Gžibovskis -
tony＠softins.co.uk -
Łukasz Posadowski