I've just finished installing a new Bacula storeage server. Prior to doing the install I did some research and ended up deciding to do the following config.
6x4TB drives /boot/efi efi_fs sda1 /boot/efi_copy efi_fs sdb1 /boot xfs RAID1 sda2 sdb2 VG RAID6 all drives containing SWAP / /home /var/bacula
Questions:
1) The big problem with this is that it is dependant on sda for booting. I did find an aritcle on how to set up boot loading on multiple HDD's, including cloning /boot/efi but I now can't find it. Does anyone know of a similar article?
2) is putting SWAP in a RAID a good idea? Will it help, will it cause problems?
I've just finished installing a new Bacula storeage server. Prior to doing the install I did some research and ended up deciding to do the following config.
6x4TB drives /boot/efi efi_fs sda1 /boot/efi_copy efi_fs sdb1 /boot xfs RAID1 sda2 sdb2 VG RAID6 all drives containing SWAP / /home /var/bacula
Questions:
- The big problem with this is that it is dependant on sda for booting.
I did find an aritcle on how to set up boot loading on multiple HDD's, including cloning /boot/efi but I now can't find it. Does anyone know of a similar article?
I also spent (wasted?) quite some time on this issue because I couldn't believe things don't work so nice with EFI as they did before. The designers of EFI obviously forgot that some people might want to boot from software RAID in a redundant way.
I ended up with a similar design than you, my fstab has this: /dev/md0 /boot xfs defaults 0 0 /dev/nvme0n1p1 /boot/efi vfat umask=0077,shortname=winnt 0 0 /dev/nvme1n1p1 /boot/efi.backup vfat umask=0077,shortname=winnt 0 0
Then in my package update tool I have a hook which syncs like this:
EFISRC="/boot/efi" EFIDEST="${EFISRC}.backup"
efisync() { if [ -d "${EFISRC}/EFI" -a -d "${EFIDEST}/EFI" ]; then rsync --archive --delete --verbose "${EFISRC}/EFI" "${EFIDEST}/" fi }
BTW, another method could be to put /boot/efi on RAID1 with metadata version 1.0 but that doesn't seem to be reliable, it works for some systems but fails on others according to report I read.
- is putting SWAP in a RAID a good idea? Will it help, will it cause
problems?
No problem at all and I don't want to lose a swap device if a disk fails. So it's the correct way to put in on RAID, IMHO.
Regards, Simon
On 1/9/19 2:30 AM, Gary Stainburn wrote:
- The big problem with this is that it is dependant on sda for booting. I
did find an aritcle on how to set up boot loading on multiple HDD's, including cloning /boot/efi but I now can't find it. Does anyone know of a similar article?
Use RAID1 for /boot/efi as well. The installer should get the details right.
- is putting SWAP in a RAID a good idea? Will it help, will it cause
problems?
It'll be moderately more reliable. If you have swap on a non-redundant disk and the kernel tries to read it, bad things (TM) will happen.
The only "drawback" that I'm aware of is that RAID consistency checks become meaningless, because it's common for swap writes to be canceled before complete, in which case one disk will have the page written but the other won't. This is by design, and considered the optimal operation. However, consistency checks don't exclude blocks used for swap, and they'll typically show mismatched blocks.
On 2019-01-09, Gordon Messmer gordon.messmer@gmail.com wrote:
On 1/9/19 2:30 AM, Gary Stainburn wrote:
- is putting SWAP in a RAID a good idea? Will it help, will it cause
problems?
The only "drawback" that I'm aware of is that RAID consistency checks become meaningless, because it's common for swap writes to be canceled before complete, in which case one disk will have the page written but the other won't. This is by design, and considered the optimal operation. However, consistency checks don't exclude blocks used for swap, and they'll typically show mismatched blocks.
If the swap is RAID1 on its own partitions (e.g., sda5/sdb5), then CHECK_DEVS in /etc/sysconfig/raid-check can be configured to check only specific devices.
--keith
On 1/9/19 2:30 AM, Gary Stainburn wrote:
- The big problem with this is that it is dependant on sda for booting.
I did find an aritcle on how to set up boot loading on multiple HDD's, including cloning /boot/efi but I now can't find it. Does anyone know of a similar article?
Use RAID1 for /boot/efi as well. The installer should get the details right.
Are you sure? How is the EFI firmware going to know about the RAID1?
RAID1 is going to have type FD00 (Linux RAID) whereas EFI firmware expects type EF00 (EFI System Partition) to boot from.
EFI then reads the GPT table directly and looks for a vfat filesystem to boot from. Default Linux software RAID1 on EL7 uses metadata 1.2 which is located at the beginning of the partition. EFI won't recognize the vfat filesystem behind the RAID metadata.
Maybe certain EFI firmware is more tolerant but at least in my case I didn't get it to work on RAID1 at all.
I'd really be interested if someone got it to work, how exactly it's configured. How exactly do the GPT tables look, how exactly is the RAID1 configured.
Regards, Simon
On Thu, 10 Jan 2019 at 11:35, Simon Matter via CentOS centos@centos.org wrote:
On 1/9/19 2:30 AM, Gary Stainburn wrote:
- The big problem with this is that it is dependant on sda for booting.
I did find an aritcle on how to set up boot loading on multiple HDD's, including cloning /boot/efi but I now can't find it. Does anyone know of a similar article?
Use RAID1 for /boot/efi as well. The installer should get the details right.
Are you sure? How is the EFI firmware going to know about the RAID1?
RAID1 is going to have type FD00 (Linux RAID) whereas EFI firmware expects type EF00 (EFI System Partition) to boot from.
It works on our Dell and IBM systems. I have multiple systems with multiple disk RAID1 /boot/efi disks.
part raid.300 --fstype="mdmember" --ondisk=sda --size=477 part raid.310 --fstype="mdmember" --ondisk=sdb --size=477 part raid.320 --fstype="mdmember" --ondisk=sdc --size=477 part raid.330 --fstype="mdmember" --ondisk=sdd --size=477 part raid.340 --fstype="mdmember" --ondisk=sde --size=477 part raid.350 --fstype="mdmember" --ondisk=sdf --size=477 part raid.360 --fstype="mdmember" --ondisk=sdg --size=477 part raid.370 --fstype="mdmember" --ondisk=sdh --size=477
raid /boot/efi --device=1 --fstype="efi" --level=RAID1 --fsoptions="umask=0077,shortname=winnt" raid.300 raid.310 raid.320 raid.330 raid.340 raid.350 raid.360 raid.370
That said, I don't know what the low level items for it are or if this is universal. From dealing with various EFI chipsets.. I am guessing the standard is rather loose in what is EFI compliant :).
EFI then reads the GPT table directly and looks for a vfat filesystem to boot from. Default Linux software RAID1 on EL7 uses metadata 1.2 which is located at the beginning of the partition. EFI won't recognize the vfat filesystem behind the RAID metadata.
Maybe certain EFI firmware is more tolerant but at least in my case I didn't get it to work on RAID1 at all.
I'd really be interested if someone got it to work, how exactly it's configured. How exactly do the GPT tables look, how exactly is the RAID1 configured.
Regards, Simon
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
On 1/10/19 8:35 AM, Simon Matter via CentOS wrote:
Are you sure?
Yes.
How is the EFI firmware going to know about the RAID1?
It doesn't specifically. Anaconda will create two EFI boot entries, each referring to one of the mirror components:
# efibootmgr -v BootCurrent: 0001 Timeout: 1 seconds BootOrder: 0001,0000 Boot0000* CentOS Linux HD(1,GPT,534debcc-f3d6-417a-b5d4-10b4ba5c1f7d,0x800,0x5f000)/File(\EFI\CENTOS\SHIM.EFI) Boot0001* CentOS Linux HD(1,GPT,05883d1e-df21-4938-bfa1-ac11ff376572,0x800,0x5f000)/File(\EFI\CENTOS\SHIM.EFI) # blkid -p -o value -s PART_ENTRY_UUID /dev/sda1 534debcc-f3d6-417a-b5d4-10b4ba5c1f7d # blkid -p -o value -s PART_ENTRY_UUID /dev/sdb1 05883d1e-df21-4938-bfa1-ac11ff376572
RAID1 is going to have type FD00 (Linux RAID) whereas EFI firmware expects type EF00 (EFI System Partition) to boot from.
I think you're referring to MBR partition types. I'm not certain, but I don't see those values in a GPT.
EFI then reads the GPT table directly and looks for a vfat filesystem to boot from. Default Linux software RAID1 on EL7 uses metadata 1.2 which located at the beginning of the partition. EFI won't recognize the vfat filesystem behind the RAID metadata.
Anaconda knows that it needs to use 1.0 metadata for EFI system partitions, which is what I mean when I said "the installer should get the details right."
# df /boot/efi/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/md125 194284 11300 182984 6% /boot/efi # mdadm --detail /dev/md125 /dev/md125: Version : 1.0
Maybe certain EFI firmware is more tolerant but at least in my case I didn't get it to work on RAID1 at all.
I'd really be interested if someone got it to work, how exactly it's configured. How exactly do the GPT tables look, how exactly is the RAID1 configured.
It might be more useful to know if this is still a problem for you with the current release. I'm not sure when support for this was added to Anaconda, but it was fairly recent. You may have last tried this before that release.
It doesn't specifically. Anaconda will create two EFI boot entries, each referring to one of the mirror components:
# efibootmgr -v BootCurrent: 0001 Timeout: 1 seconds BootOrder: 0001,0000 Boot0000* CentOS Linux HD(1,GPT,534debcc-f3d6-417a-b5d4-10b4ba5c1f7d,0x800,0x5f000)/File(\EFI\CENTOS\SHIM.EFI) Boot0001* CentOS Linux HD(1,GPT,05883d1e-df21-4938-bfa1-ac11ff376572,0x800,0x5f000)/File(\EFI\CENTOS\SHIM.EFI) # blkid -p -o value -s PART_ENTRY_UUID /dev/sda1 534debcc-f3d6-417a-b5d4-10b4ba5c1f7d # blkid -p -o value -s PART_ENTRY_UUID /dev/sdb1 05883d1e-df21-4938-bfa1-ac11ff376572
So far my config looks similar.
RAID1 is going to have type FD00 (Linux RAID) whereas EFI firmware expects type EF00 (EFI System Partition) to boot from.
I think you're referring to MBR partition types. I'm not certain, but I don't see those values in a GPT.
No, it's GPT. What does "gdisk -l /dev/sda" show? That would be interesting to see.
EFI then reads the GPT table directly and looks for a vfat filesystem to boot from. Default Linux software RAID1 on EL7 uses metadata 1.2 which located at the beginning of the partition. EFI won't recognize the vfat filesystem behind the RAID metadata.
Anaconda knows that it needs to use 1.0 metadata for EFI system partitions, which is what I mean when I said "the installer should get the details right."
# df /boot/efi/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/md125 194284 11300 182984 6% /boot/efi # mdadm --detail /dev/md125 /dev/md125: Version : 1.0
OK I see. Do you also have /boot mounted on its own MD device?
Maybe certain EFI firmware is more tolerant but at least in my case I didn't get it to work on RAID1 at all.
I'd really be interested if someone got it to work, how exactly it's configured. How exactly do the GPT tables look, how exactly is the RAID1 configured.
It might be more useful to know if this is still a problem for you with the current release. I'm not sure when support for this was added to Anaconda, but it was fairly recent. You may have last tried this before that release.
IIRC the initial install was with 7.5.
Thanks, Simon
On 1/10/19 10:20 AM, Simon Matter via CentOS wrote:
RAID1 is going to have type FD00 (Linux RAID) whereas EFI firmware expects type EF00 (EFI System Partition) to boot from.
I think you're referring to MBR partition types. I'm not certain, but I don't see those values in a GPT.
No, it's GPT. What does "gdisk -l /dev/sda" show? That would be interesting to see.
If you're looking for the "code" column, they're all 0700. See the man page for gdisk. The section describing the "l" command clarifies:
For ease of data entry, gdisk compresses these into two-byte (four-digit hexadecimal) values that are related to their equivalent MBR codes.
... Note that these two-byte codes are unique to gdisk.
The codes printed by gdisk don't actually appear in the GPT.
OK I see. Do you also have /boot mounted on its own MD device?
Yes.
I completed the install as described below which went well.
I then copied the contents of /boot/efi to /boot/efi_copy using
cd /boot rsync -acr efi/ efi_copy
I then installed grub2 on sdb
yum install yum install grub2-efi-modules grub2-install /dev/sdb
everything ran as expected with no errors. The instructions then said to test the install by powering off and swapping sda and sdb. Instead, I just unplugged sda.
When I booted the system loaded straight into a grub> prompt.
I powered off and reconnected sda hoping to be able to continue experimenting. However, even with sda reconnected it just boots into a grub> prompt.
Anyone got suggestions what to do next?
(This server is not live, so can be wiped / experimented on)
On Wednesday 09 January 2019 10:30:54 Gary Stainburn wrote:
I've just finished installing a new Bacula storeage server. Prior to doing the install I did some research and ended up deciding to do the following config.
6x4TB drives /boot/efi efi_fs sda1 /boot/efi_copy efi_fs sdb1 /boot xfs RAID1 sda2 sdb2 VG RAID6 all drives containing SWAP / /home /var/bacula
Questions:
- The big problem with this is that it is dependant on sda for booting. I
did find an aritcle on how to set up boot loading on multiple HDD's, including cloning /boot/efi but I now can't find it. Does anyone know of a similar article?
- is putting SWAP in a RAID a good idea? Will it help, will it cause
problems? _______________________________________________ CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
I completed the install as described below which went well.
I then copied the contents of /boot/efi to /boot/efi_copy using
cd /boot rsync -acr efi/ efi_copy
I then installed grub2 on sdb
yum install yum install grub2-efi-modules grub2-install /dev/sdb
everything ran as expected with no errors. The instructions then said to test the install by powering off and swapping sda and sdb. Instead, I just unplugged sda.
When I booted the system loaded straight into a grub> prompt.
I powered off and reconnected sda hoping to be able to continue experimenting. However, even with sda reconnected it just boots into a grub> prompt.
Anyone got suggestions what to do next?
If you boot, istn't there a way to get into an EFI boot menu where you can see the different EFI boot entries? Maybe you can see there what is wrong now.
On the other hand, maybe you could even reinstall and try to configure /boot/efi on RAID1 as it was described by others in this thread?
Regards, Simon
On Friday 11 January 2019 15:16:12 Simon Matter via CentOS wrote:
If you boot, istn't there a way to get into an EFI boot menu where you can see the different EFI boot entries? Maybe you can see there what is wrong now.
On the other hand, maybe you could even reinstall and try to configure /boot/efi on RAID1 as it was described by others in this thread?
I have gone down this route, and re-installed. This time I did do the EFI partition as RAID1 as people have suggested and this time the installer let me select what I wanted without warnings.
We'll see what happens after the install completes, which unfortunately will probably be after I finish tonight.
On 1/11/19 3:22 AM, Gary Stainburn wrote:
I then installed grub2 on sdb
yum install yum install grub2-efi-modules grub2-install /dev/sdb
If you want to manually provide redundant EFI system partitions (no RAID1), all you need to do is run efibootmgr to add a second entry. You don't need to install grub2-efi-modules or run grub2-install. Use:
efibootmgr -c -w -L 'CentOS Linux' -d /dev/sdb -p 1 -l 'EFI\centos\shim.efi'
... where the option to -d is the name of the drive containing the EFI system partition and the option to -p is the EFI system partition number.
For reference, that command will appear in /var/log/anaconda/program.log after any normal installation.
-
Gary Stainburn -
Gordon Messmer -
Keith Keller -
Simon Matter -
Stephen John Smoogen