Hi all!
I'm still on C6. I'm using a RAID1 configuration (Linux software RAID) and I'd like to either use the same one, or possibly configure it on new drives (larger) when I upgrade to C7. (I'm really feeling the need to move off C6.)
But it isn't at all obvious how one would do a new RAID1 setup in Anaconda, and I don't find any user reports or other info on this in the WIKI or forums (fora, properly, if I recall my high school Latin).
Can anyone provide (or give pointers to) a good recipe for doing this?
thanks!
Fred
--On Wednesday, November 18, 2015 11:07:12 PM -0500 Fred Smith fredex@fcshome.stoneham.ma.us wrote:
But it isn't at all obvious how one would do a new RAID1 setup in Anaconda
Don't feel bad. The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers. Oh, the irony.
Can anyone provide (or give pointers to) a good recipe for doing this?
A quick google brought up the following link that (looking just at the disk portion) appears to be mostly correct, and should give you the magic incantation:
http://www.ictdude.com/howto/install-centos-7-software-raid-lvm/
The one thing I would point out regarding the above link is that despite conventional UNIX wisdom, *don't* put /usr on a separate filesystem in CentOS 7. <sarcasm>Thank you RedHat</sarcasm>
Flames to /dev/null.
Devin
On Wed, Nov 18, 2015 at 09:49:43PM -0700, Devin Reade wrote:
--On Wednesday, November 18, 2015 11:07:12 PM -0500 Fred Smith fredex@fcshome.stoneham.ma.us wrote:
But it isn't at all obvious how one would do a new RAID1 setup in Anaconda
Don't feel bad. The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers. Oh, the irony.
Can anyone provide (or give pointers to) a good recipe for doing this?
A quick google brought up the following link that (looking just at the disk portion) appears to be mostly correct, and should give you the magic incantation:
http://www.ictdude.com/howto/install-centos-7-software-raid-lvm/
The one thing I would point out regarding the above link is that despite conventional UNIX wisdom, *don't* put /usr on a separate filesystem in CentOS 7. <sarcasm>Thank you RedHat</sarcasm>
Flames to /dev/null.
Devin
that looks great, thank you!
Fred Smith wrote:
On Wed, Nov 18, 2015 at 09:49:43PM -0700, Devin Reade wrote:
--On Wednesday, November 18, 2015 11:07:12 PM -0500 Fred Smith fredex@fcshome.stoneham.ma.us wrote:
But it isn't at all obvious how one would do a new RAID1 setup in Anaconda
Don't feel bad. The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers. Oh, the irony.
Agreed. What I do, all the time for a new server, is to get to the screen in the graphical install where you select a drive, then <alt><f2>, and do what I need to - partition, make a RAID, whatever, then switch back to the graphical, <done> and now set mount points, etc.
mark
Fred Smith wrote:
On Wed, Nov 18, 2015 at 09:49:43PM -0700, Devin Reade wrote:
--On Wednesday, November 18, 2015 11:07:12 PM -0500 Fred Smith fredex@fcshome.stoneham.ma.us wrote:
But it isn't at all obvious how one would do a new RAID1 setup in Anaconda
Don't feel bad. The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers. Oh, the irony.
<snip> Right, hit <send>, then thought of one more annoyance: the stupid graphical custom partition option will NOT do what you tell it: I want /boot first, then swap, *then* /, but if you don't do it the way I mentioned in my last post - <alt><f2> and parted, it renumbers and reassigns your partitions Magically <fairy dust/>, and puts / after /boot, and before swap. For no particular reason, and with no explanation.
mark
On Thu, November 19, 2015 8:15 am, m.roth@5-cent.us wrote:
Fred Smith wrote:
On Wed, Nov 18, 2015 at 09:49:43PM -0700, Devin Reade wrote:
--On Wednesday, November 18, 2015 11:07:12 PM -0500 Fred Smith fredex@fcshome.stoneham.ma.us wrote:
But it isn't at all obvious how one would do a new RAID1 setup in Anaconda
Don't feel bad. The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers. Oh, the irony.
<snip> Right, hit <send>, then thought of one more annoyance: the stupid graphical custom partition option will NOT do what you tell it: I want /boot first, then swap, *then* /, but if you don't do it the way I mentioned in my last post - <alt><f2> and parted, it renumbers and reassigns your partitions Magically <fairy dust/>, and puts / after /boot, and before swap. For no particular reason, and with no explanation.
Indeed, make something that even idiot can use, and only idiot will [be happy to] use it.
My C7 boxes (workstations, no C7 servers, please) are all installed using kickstart, and drives are partitioned exactly as I described in kickstart file. There is still something we can happily use ;-)
Valeri
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 11/19/2015 06:56 AM, Valeri Galtsev wrote:
My C7 boxes (workstations, no C7 servers, please) are all installed using kickstart, and drives are partitioned exactly as I described in kickstart file. There is still something we can happily use;-)
Perhaps by luck. Anaconda will definitely reorder partitions if you describe them with the "part" command in kickstart. The *only* way to get predictable partition layouts is to manually partition.
On Fri, November 20, 2015 10:49 am, Gordon Messmer wrote:
On 11/19/2015 06:56 AM, Valeri Galtsev wrote:
My C7 boxes (workstations, no C7 servers, please) are all installed using kickstart, and drives are partitioned exactly as I described in kickstart file. There is still something we can happily use;-)
Perhaps by luck. Anaconda will definitely reorder partitions if you describe them with the "part" command in kickstart. The *only* way to get predictable partition layouts is to manually partition.
And indeed it did. Not to the extent of going ultimately against my scheme. OK, CentOS (7) just fell yet one more notch down in my opinion. Thanks for correcting me.
Valeri
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On 11/20/2015 09:11 AM, Valeri Galtsev wrote:
And indeed it did. Not to the extent of going ultimately against my scheme. OK, CentOS (7) just fell yet one more notch down in my opinion. Thanks for correcting me.
Have a look at the links I sent earlier and see if that approach might work for you. If not, I'd be interested in what needs it doesn't meet.
On Nov 18, 2015, at 9:49 PM, Devin Reade gdr@gno.org wrote:
The one thing I would point out regarding the above link is that despite conventional UNIX wisdom, *don't* put /usr on a separate filesystem in CentOS 7. <sarcasm>Thank you RedHat</sarcasm>
Flames to /dev/null.
Sorry, you don’t get to throw that grenade and then run away.
The old wisdom you refer to is simply obsolete, and it wasn’t Red Hat that made it so. ’Twas *progress* that made it so, specifically the fact that even a throwaway USB key has enough space to hold the complete OS on it these days. We no longer live in a world of 5 MB disk packs the size of extra large pizzas.
Several OSes made /usr/{bin,lib} the same as /{bin,lib} way before Red Hat: Solaris, OS X, and Cygwin, at the least. Probably all of the embedded Linuxes, too.
Even FreeBSD is starting to give up on /usr as separate from /. Although its installer still lets you put /usr into a separate slice, the boot process will break if you put it onto a different physical disk that the boot kernel can’t see, or use a different filesystem for it that isn’t compiled into the boot kernel.
A truly “traditional” Unix OS wouldn’t have this problem: as long as the tools necessary to mount /usr are in /, it would be able to boot.
If you disbelieve that, try installing FreeBSD 9.0 (i.e. pre-ZFS-boot) into a VM, then move /usr into a ZFS pool. It won’t even boot into single-user mode! Ask me how I know.
On Nov 19, 2015, at 11:54 AM, Warren Young wyml@etr-usa.com wrote:
Several OSes made /usr/{bin,lib} the same as /{bin,lib} way before Red Hat: Solaris, OS X, and Cygwin, at the least.
Small correction: OS X doesn’t symlink /{bin,lib} to /usr/{bin,lib} as Solaris, EL7, and Cygwin do, but the installer doesn’t offer the option to put it onto a separate volume/partition, and I doubt the OS would boot if you did later move it.
--On Thursday, November 19, 2015 11:54:19 AM -0700 Warren Young wyml@etr-usa.com wrote:
?Twas *progress* that made it so, specifically the fact that even a throwaway USB key has enough space to hold the complete OS on it these days.
For the record, size of the disk was never the original motivation for keeping / separate, at least within my memory; it was to minimize the amount of disk space that needed to be fsck'd before bringing the system to single user mode without needing to revert to "rescue media" (at that time, the 125MB tape on which the OS was shipped). Even in the days of SunOS 3 (that's SunOS, not Solaris) I was installing the entire OS on one physical drive, partitioned. Sure, NFS and USENET servers had many drives, but most machines were single drives.
Beyond that, I don't really want to get into a flame war on stuff that has been hashed through before.
The main message is that while CentOS 6 and before *could* have / and /usr on different filesystems, don't do it with CentOS 7 per https://access.redhat.com/solutions/53005 (paywalled). TL;DR: you'll have trouble booting some configurations including those using iSCSI.
Devin
On Thu, November 19, 2015 4:49 pm, Devin Reade wrote:
--On Thursday, November 19, 2015 11:54:19 AM -0700 Warren Young wyml@etr-usa.com wrote:
?Twas *progress* that made it so, specifically the fact that even a throwaway USB key has enough space to hold the complete OS on it these days.
For the record, size of the disk was never the original motivation for keeping / separate, at least within my memory; it was to minimize the amount of disk space that needed to be fsck'd before bringing the system to single user mode without needing to revert to "rescue media"
Not only. Separation of / /var /usr /home (/var/mail, ...) also was a protection from local user [accidental] denial of service attack by preventing him from filling up or running out of inodes these filesystems.
Valeri
(at that time, the 125MB tape on which the OS was shipped). Even in the days of SunOS 3 (that's SunOS, not Solaris) I was installing the entire OS on one physical drive, partitioned. Sure, NFS and USENET servers had many drives, but most machines were single drives.
Beyond that, I don't really want to get into a flame war on stuff that has been hashed through before.
The main message is that while CentOS 6 and before *could* have / and /usr on different filesystems, don't do it with CentOS 7 per https://access.redhat.com/solutions/53005 (paywalled). TL;DR: you'll have trouble booting some configurations including those using iSCSI.
Devin
CentOS mailing list CentOS@centos.org https://lists.centos.org/mailman/listinfo/centos
++++++++++++++++++++++++++++++++++++++++ Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247 ++++++++++++++++++++++++++++++++++++++++
On Nov 19, 2015, at 3:49 PM, Devin Reade gdr@gno.org wrote:
size of the disk was never the original motivation for keeping / separate, at least within my memory
Prior to SysV, the location of user home directories was not standardized, and AT&T recommended that you put them in /usr.[1]
Also, in the PDP days, you had things like the RL and RK series drives, which started at 2.5 MB. If you were well-heeled, you might have several of these, getting you up into the tens of megs. Thus, it made sense to put one major filesystem on each disk, like /, /usr, and /var for a 3-drive system.
it was to minimize the amount of disk space that needed to be fsck'd before bringing the system to single user mode
That may have been a happy side benefit, but I’ve never read such a thing in a quarter century of using Unix.
Reference?
Even in the days of SunOS 3 (that's SunOS, not Solaris) I was installing the entire OS on one physical drive, partitioned.
Unix goes back about 15 years before the advent of the Sun-3. Being fairly high-end boxes of the time, they likely would have had HDDs in the hundreds of megs range, possibly even the single-digit gigs, plenty for a complete 4.2BSD install.
This was also the time in computing history where diskless or small-disk workstations were common, which gives another reason to make /usr separate: get an NFS stack up via RARP or the small local system drive, then attach to a server’s disk to complete the boot process.
The main message is that while CentOS 6 and before *could* have / and /usr on different filesystems, don't do it with CentOS 7 per https://access.redhat.com/solutions/53005 (paywalled).
I don’t need to pierce the paywall to know it’s a bad idea. This is enough of a clue:
$ ls -l /bin lrwxrwxrwx. 1 root root 7 Jul 7 2014 /bin -> usr/bin
[1]: See page 4-8 in the SVR3.2 sysadmin’s guide: http://goo.gl/E9quko
On Nov 19, 2015, at 9:02 PM, Warren Young wyml@etr-usa.com wrote:
Prior to SysV…
Sorry, that should be “SVR4”.
On 11/18/2015 08:49 PM, Devin Reade wrote:
The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers.
I really don't think that's an appropriate or useful way to discuss software.
On Wed, Nov 18, 2015 at 09:49:43PM -0700, Devin Reade wrote:
--On Wednesday, November 18, 2015 11:07:12 PM -0500 Fred Smith fredex@fcshome.stoneham.ma.us wrote:
But it isn't at all obvious how one would do a new RAID1 setup in Anaconda
Don't feel bad. The abortion that is the RHEL/CentOS 7 graphical install interface is far too dumbed-down to be easily usable by anyone that understands what is going on under the covers. Oh, the irony.
Can anyone provide (or give pointers to) a good recipe for doing this?
A quick google brought up the following link that (looking just at the disk portion) appears to be mostly correct, and should give you the magic incantation:
http://www.ictdude.com/howto/install-centos-7-software-raid-lvm/
I've been building C7 VMs with RAID-1 in Virtualbox these last couple of days, to pin down exactly how to do it. based on the link above, but I'm making "real" partitions instead of LVM.
The current iteration seems to be running fine, but I had an odd problem when configuring the partitions: for two 10 gig virtual drives, it wouldn't let me use the last gig of space. or maybe it has some other issue on the max partition size,... it seemed to max out at around 9900 MB. This for sure won't do when I build it on real hardware, so I'm wondering if anyone else here has a clue what's going on??
thanks in advance!
On 11/24/2015 07:35 PM, Fred Smith wrote:
it seemed to max out at around 9900 MB. This for sure won't do when I build it on real hardware, so I'm wondering if anyone else here has a clue what's going on??
IIRC, Anaconda can't create "raid1" LVM volumes. That is, you can't create LVs that are mirrored across multiple PVs. You can create md RAID1 volumes and make a PV of the resulting RAID volume, though.
So I think what you're probably doing is creating 100MB /boot partition, and then two md RAID component partitions of 9900MB.
On Tue, Nov 24, 2015 at 10:32:16PM -0800, Gordon Messmer wrote:
On 11/24/2015 07:35 PM, Fred Smith wrote:
it seemed to max out at around 9900 MB. This for sure won't do when I build it on real hardware, so I'm wondering if anyone else here has a clue what's going on??
IIRC, Anaconda can't create "raid1" LVM volumes. That is, you can't create LVs that are mirrored across multiple PVs. You can create md RAID1 volumes and make a PV of the resulting RAID volume, though.
So I think what you're probably doing is creating 100MB /boot partition, and then two md RAID component partitions of 9900MB.
yeah, you're probably right. but for some reason, Anaconda is still telling me there's a gigabyte of unused space remaining (105x megabytes where i've forgotten what the "x" was.) Might that be because I didn't put swap on both drives (and I think I set up swap as a gig)?
should I put swap on both drives? should it be RAID1 also, or just two swap partitions not in any RAID volume?
(I remember seeing a somewhat technical article in the last couple of years that talked of a potential corruption issue if one puts swap into a Linux software RAID-1 volume, but no longer recall the details.)
On 11/25/2015 09:09 AM, Fred Smith wrote:
should I put swap on both drives? should it be RAID1 also, or just two swap partitions not in any RAID volume?
I really recommend using the fewest partitions possible, since md RAID operates on partitions (Anaconda won't create full disk md RAID volumes), and replacing a disk will require you to handle each partition individually. It's probably best to do a small /boot on RAID1 and use the rest of the disk for a second RAID1 volume, with LVM on that.
(If you haven't looked at the links I posted earlier, do consider that)
(I remember seeing a somewhat technical article in the last couple of years that talked of a potential corruption issue if one puts swap into a Linux software RAID-1 volume, but no longer recall the details.)
There's no corruption. If a write is canceled before it completes, you might end up with a block mis-match (or might have, I'm not sure if that still happens), but no corruption. The mis-match is on a block that isn't mapped to anything, and will never be read by anything other than a RAID verify. It's ugly because it means the verify operation is no longer useful, but it won't cause any read from the RAID volume to return bad data.
On Wed, 25 Nov 2015, Gordon Messmer wrote:
I really recommend using the fewest partitions possible,
replacing a disk will require you to handle each partition individually.
This is not a large burden, but I do agree to keep it simple.
It's probably best to do a small /boot on RAID1 and use the rest of the disk for a second RAID1 volume, with LVM on that.
The default configuration is a partition for /boot and one per drive for LVM PVs all of which are used by a single VG, in which LVs for / (which has a fixed upper size), swap (which uses a computed size) and if there's space available /home (which uses all remaining space).
And there's the problem, the installer uses all available space leaving none for a RAID of /boot/.
I believe this is due to an assumption that Red Hat makes, that their customers are mostly enterprise and thus the drives that the installer sees would be logical and backed by RAID.
I could wish that the installer would RAID /boot by default, which is small enough that it is unlikely to be a burden for those with RAID backed "drives" and a boon to those who do not.
What's needed is enough space for the installer to carve out additional partitions on the additional drives. Which means manual partitioning, and in particular you must delete or resize /home -- remember, this is in the initial configuration so there's nothing to lose, and you can create/resize it a little later anyway. If you have very small disks you probably need to shrink / and/or swap.
Reduce the space to be used by enough for another /boot and you can change /boot's device type to one of the RAIDs.
Reduce it to no more than what the first drive provides and you can also setup RAID under the PV -- modify the volume group redundancy.
After which you can make mount points for anything else, e.g., /var/www, and grow what you shrank.
If you like pain you can back into such a layout after installation is complete, but I do not recommend it.
When there are more than 2 drives possible I prefer MD RAID for /boot and LVM RAID for everything else -- the not likeable part is that even though LVM RAID leverages MD RAID the status is not presented via /proc/mdstat, which changes how one must monitor for failures. When only 2 drives are possible odds are you want RAID1 and it might as well be under the PV.
Using LVM RAID is interesting in that it allows you to decide later without much effort the redundancy you want on a per mount point basis, e.g., on a 4 drive system it is just two commands to create mount points (LVs) using RAID10 for /var/lib/*sql but RAID5 for /var/www -- that's not impossible to do with MD RAID under PVs but it probably means lots more commands and even some juggling.
/mark
Fred Smith wrote:
On Wed, Nov 18, 2015 at 09:49:43PM -0700, Devin Reade wrote:
--On Wednesday, November 18, 2015 11:07:12 PM -0500 Fred Smith fredex@fcshome.stoneham.ma.us wrote:
<snip>
I've been building C7 VMs with RAID-1 in Virtualbox these last couple of days, to pin down exactly how to do it. based on the link above, but I'm making "real" partitions instead of LVM.
The current iteration seems to be running fine, but I had an odd problem when configuring the partitions: for two 10 gig virtual drives, it wouldn't let me use the last gig of space. or maybe it has some other issue on the max partition size,... it seemed to max out at around 9900 MB. This for sure won't do when I build it on real hardware, so I'm wondering if anyone else here has a clue what's going on??
Two thoughts: one, as Gordon suggested, it might be making a /boot partition. The other thought is that it might be trying to align the partition optimal. Still, that's usually 1M or 2M, not 1G.
Btw, when I manually create partitions, with parted, I use mkpart pri 0.0GB 100% (for data drives). Assuming I've gone in with parted -a opt, it does the right thing. Hope this gives you some ideas.
mark
On 11/18/2015 08:07 PM, Fred Smith wrote:
Can anyone provide (or give pointers to) a good recipe for doing this?
One of the things I'm working on right now is moving toward a single standard partition layout for all systems, with RAID or without. The reason I want to do that is that it'll allow me to script the replacement of a failed drive, both so that I don't fat-finger it and so that our on-call staff can reasonably expect to replace a drive.
In order to do that, I set up GPT on all drives, regardless of whether they are used in BIOS systems or UEFI. On BIOS systems, the first partition is a bios_boot partition, and on UEFI, it's an EFI system partition. The next partition is a small slice for RAID1 /boot. The last partition is a member of whatever RAID level is appropriate for the system.
There are a couple of limiting factors on that. First, Anaconda won't create a large partition for bios_boot. Second, even in kickstart, Anaconda will reorder partitions. So, I can't use Anaconda to create partitions and achieve a uniform layout.
I use this kickstart file for initial setup: http://home.dragonsdawn.net/kickstart/centos7
And this script to replace drives: http://home.dragonsdawn.net/kickstart/add-raid-member
I still haven't decided how I'm going to handle EFI system partitions in the replacement script and otherwise, so this system is incomplete. But if people want to take a look and comment, I'd be interested in discussion.
-
Devin Reade -
Fred Smith -
Gordon Messmer -
m.roth＠5-cent.us -
Mark Milhollan -
Valeri Galtsev -
Warren Young