I have a CentOS 4.2 system that was set up VERY quickly following the demise of its former life as a CentOS 3 server - you don't want the full story, but it had to be done quickly to get a company up and working following a slight disaster involving an electrician, a portable appliance safety tester and a pulled power cable - anyway, here's where I am at...
Everything is running fine but I now want to mirror the root partition (/dev/hda3). I have read quite a few set of notes on the theory but since they contain conflicting advice I am hoping someone here can set me straight - here's what IO want to do:
hda3 and hde3 become a mirrored pair (/dev/md0) and this is mounted at boot time as /
Is it as simple as a quick mdadm command and then an edit of /etc/inittab??
If I try and use mdadm right now I get complaints that /dev/hda3 is busy (fair enough) so I guess I need to be in less than 'full steam ahead' mode to make the change - over to you....
Thanks
Once you've mdadm and done you're other stuff you can use grub, to install a boot loader on each disk, if you are using grub...
You might find this excerpt useful:
============================ LinuxSA Mailing list archives Index: [thread] [date] [subject] [author] [stats] --------------------------------------------------------------------------------
From: Michael Martucci m.martucci@gmx.net To : linuxsa@linuxsa.org.au Date: Tue, 29 Jul 2003 14:44:46 +0930
GRUB + RAID howto In the Software RAID howto it is mentioned that it is not known how to set up GRUB to boot off RAID. Here is how I did it: **Follow at your own risk. If you break something it's your fault.** ================================================================== Configuration: - /dev/hda (Pri. Master) 60 GB Seagate HDD (blank) - /dev/hdc (Sec. Master) 60 GB Seagate HDD (blank) - /dev/hdd (Sec. Slave) CDROM Drive
Setup Goals: - /boot as /dev/md0: RAID1 of /dev/hda1 & /dev/hdc1 for redundancy - / as /dev/md1: RAID1 of /dev/hda2 & /dev/hdc2 for redundancy - swap*2 with equal priority: /dev/hda3 & /dev/hdc3 for more speed - GRUB installed in boot records of /dev/hda and /dev/hdc so either drive can fail but system still boot.
Tools: - mdadm (http://www.cse.unsw.edu.au/~neilb/source/mdadm/) (I used 1.2.0, but notice that as of 20030729 1.3.0 is available)
1. Boot up off rescue/installation CD/disk/HDD/whatever with mdadm tools installed.
2. Partitioning of hard drives: (I won't show you how to do this. See: # man fdisk ; man sfdisk ) But here's how stuff was arranged: ------------------------------------------------------------------ # sfdisk -l /dev/hda
Disk /dev/hda: 7297 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0
Device Boot Start End #cyls #blocks Id System /dev/hda1 * 0+ 16 17- 136521 fd Linux raid autodetect /dev/hda2 17 7219 7203 57858097+ fd Linux raid autodetect /dev/hda3 7220 7296 77 618502+ 82 Linux swap /dev/hda4 0 - 0 0 0 Empty ------------------------------------------------------------------ To make /dev/hdc the same: ------------------------------------------------------------------ # sfdisk -d /dev/hda | sfdisk /dev/hdc ------------------------------------------------------------------ /dev/hd[ac]1 for /dev/md0 for /boot /dev/hd[ac]2 for /dev/md1 for / /dev/hd[ac]3 for 2*swap It is important to make md-to-be partitions with ID 0xFD, not 0x83.
3. Set up md devices: (both are RAID1 [mirrors]) ------------------------------------------------------------------ # mdadm --create /dev/md0 --level=1 \ --raid-devices=2 /dev/hda1 /dev/hdc1 # mdadm --create /dev/md1 --level=1 \ --raid-devices=2 /dev/hda2 /dev/hdc2 ------------------------------------------------------------------
4. Make filesystems: ------------------------------------------------------------------ # mke2fs /dev/md0 # mkreiserfs /dev/md1 # mkswap /dev/hda3 # mkswap /dev/hdc3 ------------------------------------------------------------------
5. Install Your distribution: Simply treat /dev/md0 and /dev/md1 as the partitions to install on, and install the way your normally do. Eg, for Gentoo: ------------------------------------------------------------------ # mkdir newinst # mount -t reiserfs /dev/md1 ./newinst # cd newinst # mkdir boot # mount -t ext2 /dev/md0 ./boot # tar -xvjpf ../stage1-x86-1.4_rc2.tbz2 # mount -o bind /proc ./proc # chroot ./ ... ------------------------------------------------------------------ Here're the relevant entries /etc/fstab for the newly created partitions: ------------------------------------------------------------------ /dev/md0 /boot ext2 noauto,noatime 1 1 /dev/md1 / reiserfs noatime 1 1 /dev/hda3 none swap sw,pri=1 0 0 /dev/hdc3 none swap sw,pri=1 0 0 ------------------------------------------------------------------ The "pri=1" for each of the swap partitions makes them the same priority so the kernel does striping and that speeds up vm. Of course, this means that if a disk dies then the system may crash, needing a reboot. Perhaps it would be wiser to make hd[ac]3 a RAID1 array too, and just use /dev/md2 as swap.
6. Setting up GRUB: (assuming you've already installed it) ------------------------------------------------------------------ # grub grub> root (hd0,0) Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd0) Checking if "/boot/grub/stage1" exists... yes Checking if "/boot/grub/stage2" exists... yes Checking if "/boot/grub/e2fs_stage1_5" exists... yes Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 16 sectors are embedded. succeeded Running "install /boot/grub/stage1 (hd0) (hd0)1+16 p (hd0,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded Done.
grub> root (hd1,0) Filesystem type is ext2fs, partition type 0xfd
grub> setup (hd1) Checking if "/boot/grub/stage1" exists... yes Checking if "/boot/grub/stage2" exists... yes Checking if "/boot/grub/e2fs_stage1_5" exists... yes Running "embed /boot/grub/e2fs_stage1_5 (hd1)"... 16 sectors are embedded. succeeded Running "install /boot/grub/stage1 (hd1) (hd1)1+16 p (hd1,0)/boot/grub/stage2 /boot/grub/grub.conf"... succeeded Done.
grub> quit ------------------------------------------------------------------ Here is how /boot/grub/grub.conf is: (/dev/md0 mounted as /boot) (Assuming kernel is installed as /boot/bzImage, and RAID1 support compiled into the kernel). ------------------------------------------------------------------ # Boot automatically after 30 secs. timeout 30
# By default, boot the first entry. default 0
# Fallback to the second entry. fallback 1
# For booting with disc 0 kernel title GNU/Linux (hd0,0) kernel (hd0,0)/bzImage root=/dev/md1
# For booting with disc 1 kernel, if (hd0,0)/bzImage is unreadable title GNU/Linux (hd1,0) kernel (hd1,0)/bzImage root=/dev/md1 ------------------------------------------------------------------
Now you should be able to reboot your system and play! ==================================================================
Please let me know of any errors, feedback, etc.
Michael Martucci.
On 11/15/05, Peter Farrow peter@farrows.org wrote:
LinuxSA Mailing list archives
Setup Goals:
- /boot as /dev/md0: RAID1 of /dev/hda1 & /dev/hdc1 for redundancy
- / as /dev/md1: RAID1 of /dev/hda2 & /dev/hdc2 for redundancy
- swap*2 with equal priority: /dev/hda3 & /dev/hdc3 for more speed
- GRUB installed in boot records of /dev/hda and /dev/hdc so either
drive can fail but system still boot.
Interesting that swap isn't mirrored. Wonder what happens when a drive dies, and then a particular lost bit of swap needs paging back in... I always mirror swap as well. Maybe I'm wrong.
-- Cheers,
Tony
you probably have to boot from the cd using rescue mode as well:
at the cd boot prompt:
linux rescue
this will release the "busy" devices as you can unmount them as you go...
P.
Nigel kendrick wrote:
I have a CentOS 4.2 system that was set up VERY quickly following the demise of its former life as a CentOS 3 server - you don't want the full story, but it had to be done quickly to get a company up and working following a slight disaster involving an electrician, a portable appliance safety tester and a pulled power cable - anyway, here's where I am at...
Everything is running fine but I now want to mirror the root partition (/dev/hda3). I have read quite a few set of notes on the theory but since they contain conflicting advice I am hoping someone here can set me straight
- here's what IO want to do:
hda3 and hde3 become a mirrored pair (/dev/md0) and this is mounted at boot time as /
Is it as simple as a quick mdadm command and then an edit of /etc/inittab??
If I try and use mdadm right now I get complaints that /dev/hda3 is busy (fair enough) so I guess I need to be in less than 'full steam ahead' mode to make the change - over to you....
Thanks
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
That's the ticket. I wonder if anyone has tried building a "rescue USB thumb drive" since the 1GB devices are so cheap these days. That would be great for this kind of situation.
Cheers,
Peter Farrow wrote:
you probably have to boot from the cd using rescue mode as well:
at the cd boot prompt:
linux rescue
this will release the "busy" devices as you can unmount them as you go...
P.
Nigel kendrick wrote:
I have a CentOS 4.2 system that was set up VERY quickly following the demise of its former life as a CentOS 3 server - you don't want the full story, but it had to be done quickly to get a company up and working following a slight disaster involving an electrician, a portable appliance safety tester and a pulled power cable - anyway, here's where I am at...
Everything is running fine but I now want to mirror the root partition (/dev/hda3). I have read quite a few set of notes on the theory but since they contain conflicting advice I am hoping someone here can set me straight
- here's what IO want to do:
hda3 and hde3 become a mirrored pair (/dev/md0) and this is mounted at boot time as /
Is it as simple as a quick mdadm command and then an edit of /etc/inittab??
If I try and use mdadm right now I get complaints that /dev/hda3 is busy (fair enough) so I guess I need to be in less than 'full steam ahead' mode to make the change - over to you....
That's the ticket. I wonder if anyone has tried building a "rescue USB
thumb drive" since the 1GB devices are so cheap these days. That would be great for this kind of situation.
FWIW, I'm booting my firewall from an usb pen drive with devil-linux installed, so a rescue environment should be doable. But then again, devil-linux works like a rescue environment for CentOS anyway. :)
Regards, Harald
Nigel kendrick support-lists@petdoctors.co.uk wrote:
I have a CentOS 4.2 system that was set up VERY quickly following the demise of its former life as a CentOS 3
server -
you don't want the full story, but it had to be done
quickly
to get a company up and working following a slight disaster involving an electrician, a portable appliance safety
tester
and a pulled power cable - anyway, here's where I am at...
Er, um, would RAID solve that problem? I.e., did you fry the drive or get physical errors? Or did the filesystem get toasted? I mean, drive failures are one thing, but electrical/surges are a completely other thing (and RAID might not help you there ;-).
If I try and use mdadm right now I get complaints that /dev/hda3 is busy (fair enough) so I guess I need to be in less than 'full steam ahead' mode to make the change - over to you....
As many other people have pointed out, you'll need to have the system off-line to do this. The Rescue CD has pretty much everything you need -- from mdadm to grub. There are a few HOWTOS, although I find some of their example hardware configurations to be a bit disturbing (like putting an ATAPI device as a slave to an ATA drive).
It used to be more of an issue when /boot (with the initrd) couldn't be a md device, but that no longer seems to be a constraint.
-- Bryan
P.S. I'll reiterate that a $100-125 3Ware Escalade 7006-2 (ATA) or 8006-2 (SATA) card gives you quite "piece of mind" for the price when it comes to RAID-1. And since it doesn't use block striping (unlike RAID-0, 10 or 5), you can remove the disks from the RAID-1 volume and use them without the card.
With 3Ware, you don't have the boot-time issues, you don't have the ATA chipset/drive timeout issues, and the admin tools do their job quite well (with more and more standard support software like smartd being able to monitoring 3Ware kernel message, etc... as well).
Bryan wants a bit more insight:
System was running Centos 3 and had 3 PATA drives - one containing root, boot, swap and then a mirrored pair for our data. CentOS 4 wasn't around when the server was initially setup.
The mirrored pair were running off a PCI controller with an ITE 8212 chipset which was a real pain in in the ass to get going - I recall compiling the driver and adding it to the boot image manually.
Anyway, along comes Mr Electrical Inspection and - allegedly - leaves the server well alone while testing **ALL** other equipment in the office. Now, as soon as Mr E.I. leave site everyone amazingly discovers that the server is 'down'.
When I arrive on site the server is 'hung' with no image on the monitor. I power cycled it and rebooted - had to fsck root but got an error that the RAID array was damaged and I was being dropped to a repair console (or words to that effect). At that stage absolutely everything I did to try and repair the drives resulted in an error and would not touch the drives on the 8212-based controller at all.
Now, the reason we had 3 drives in the system originally was because I could not get the system to boot from a drive on the 8212 controller and with one disk on the motherboard's controller and one on the 8212 the whole lot was unstable so we ended up as described above.
When I started to have problems re-mounting the RAID drives I just 'bit the bullet' and decided to put CentOS 4 on a spare 80GB drive I'd bought with me because I knew that the 8212 controller was supported natively and it was something I was going to do at a 'convenient' time anyway - I also guessed it would be quicker and more worthwhile than fiddling with the CentOS 3 installation. The plan was also to end up with just two disk drives in the system, as originally intended, but I didn't want to (and couldn't) mess around with the RAID pair until I had CentOS4 up and running.
The system was Running CentOS4 in about 20 mins, I mounted one of the RAID drives, checked it out, copied all wanted data onto the 'new' primary disk and got the users running their company management system as soon as possible as this was the #1 priority.
After that I reinstalled the email system (postfix, MailScanner, clamAV etc. - thanks again for the excellent walk through Johnny) and did a general tidy up knowing that I could now get back on the system from my office to tidy up (caching name server, nntp time sync and a few 'home' mappings.
That's it - all that's left to do it mirror the current (one) drive as mentioned in my original email.
Hope that's all clear. If anyone wants to comment on what I did I'd be happy to hear what you think and perhaps what I might have done differently!
Nigel
Nigel kendrick support-lists@petdoctors.co.uk wrote:
Bryan wants a bit more insight:
I'm a demanding SOB, aren't I? ;->
System was running Centos 3 and had 3 PATA drives - one containing root, boot, swap and then a mirrored pair for
our
data ... The mirrored pair were running off a PCI
controller
with an ITE 8212 chipset which was a real pain in in the
ass
to get going ...
Yep, I think I remember a discussion on that chip -- a really low-cost, but true RAID-1 hardware logic chip with just enough SRAM, ~32KiB, to store 1 block transfer.
Anyway, along comes Mr Electrical Inspection and -
allegedly -
leaves the server well alone ...
Yeah, I know how that goes. Long ago I learned to put in 12000-25000VA units in my areas after someone called out electrical one day.
I've never used the 8212, so I don't know how it is for RAID-1. I really avoid anything but 3Ware, mainly because of my past experiences/trust. But that's just me.
I keep putting off testing a NetCell SR5000, largely because it has virtually no user-space support in Linux.
That's it - all that's left to do it mirror the current (one) drive as mentioned in my original email. Hope that's all clear.
Yep, thanx for reminding me where you were coming from.
Nigel kendrick wrote:
Hope that's all clear. If anyone wants to comment on what I did I'd be happy to hear what you think and perhaps what I might have done differently!
Sounds like you did what you had to in order to get your client up and running again while under a bit of time pressure. Often times, we wind up with a suboptimal solution when faced with those circumstances, but doing things "the right way" isn't always the fastest way. 8-)
If it was me (and it isn't), I probably would have set up the 3 disks in a RAID5 array. That way, you can lose one disk and the system still hums along until you can replace the failed disk. The way you had things before, you could also lose one disk and continue humming along...just as long as it wasn't the system disk that failed. 8-) If your client has the budget, and you've got the time, you might want to consider building a new box with a 3Ware hardware RAID card and 4 disks. You can run a 3-disk RAID5 with one hot spare. So if you lose a disk in the array, it will grab the spare and automagically rebuild the array. In theory, that allows you to lose 2 of your 4 drives (as long as the 2nd doesn't go away while the rebuild is happening) without the system crashing and burning. Depending on what your space requirements are like, you could upgrade to 4 80gig SATA drives and something like a 3Ware 8506-4LP or 9500S-4LP for about $600 over the cost of a new system. CentOS 4.2 should install on a RAID5 array on either of these cards without any fuss.
Cheers,
If it was me (and it isn't), I probably would have set up the 3 disks in a
RAID5 array. That way, you can lose one disk and the system still hums along until you can replace the failed disk.
That was considered at the time but we found that any combination of mirroring using the motherboard-based controller and the plugged in one wasn't stable (unbootable or soon locked-up) - the only way we could get mirroring to work was if both drives were on the 8212-based controller. Time and budget were against us at the time we were building the original system, but later work with CentOS4 and that controller on other systems was much more successful.
On Wed, 2005-11-16 at 07:41 +0000, Nigel kendrick wrote:
That was considered at the time but we found that any combination of mirroring using the motherboard-based controller and the plugged in one wasn't stable (unbootable or soon locked-up) - the only way we could get mirroring to work was if both drives were on the 8212-based controller.
In reading Cox's notes on the 8212 awhile back, I distinctly remember that he did not have a lot of faith in using the 8212 as a "dumb" ATA controller with 2 channels. It's clearly rigged in hardware to always be used as a hardware stripe/mirror logic.
Nigel kendrick wrote:
If it was me (and it isn't), I probably would have set up the 3 disks in a
RAID5 array. That way, you can lose one disk and the system still hums along until you can replace the failed disk.
That was considered at the time but we found that any combination of mirroring using the motherboard-based controller and the plugged in one wasn't stable (unbootable or soon locked-up) - the only way we could get mirroring to work was if both drives were on the 8212-based controller. Time and budget were against us at the time we were building the original system, but later work with CentOS4 and that controller on other systems was much more successful.
In the face of all that trouble, I guess my next question would be "Did you soldier on with the troublesome chipset because you had to (for budget or other reasons?) or you just didn't have the time to source out a dedicated RAID card? I guess I'm having trouble understanding the rationale behind going to all that trouble to mirror the data, but not doing the same for the OS....when you can do both at the same time with RAID5. Having the data intact is nice, but having intact data on a dead server isn't terribly useful. 8-)
Fortunately, these controllers are becoming quite cheap and disk space is practically free compared to what it cost even a few years ago. I remember having to spend "a small fortune" about 15 years ago for a 1gig disk so I could keep a week's worth of a full Usenet feed on my news server. 8-)
Cheers,
On 11/16/05, Chris Mauritz chrism@imntv.com wrote:
Nigel kendrick wrote:
If it was me (and it isn't), I probably would have set up the 3 disks in a
RAID5 array. That way, you can lose one disk and the system still hums along until you can replace the failed disk.
I have successfully survived a hdd crash in raid1 since the system was set to boot up from second disk as well. We copied all partitions in disk one to disk two (including root) and then setup some grub magic. so that the second disk was also bootable. And this off a software raid. -- Sudev Barar Learning Linux
On Thu, 2005-11-17 at 07:53 +0530, Sudev Barar wrote:
I have successfully survived a hdd crash in raid1 since the system was set to boot up from second disk as well. We copied all partitions in disk one to disk two (including root) and then setup some grub magic. so that the second disk was also bootable. And this off a software raid.
Before I started using 3Ware, I had a Lilo script that did this on the second drive with a redundant /boot[.1] filesystem.
Of course, it wasn't true failover. I had to disable the other disk for it to work, or the controller board had to fail so it didn't appear.
In the face of all that trouble, I guess my next question would be "Did you
soldier on with the troublesome chipset
because you had to (for budget or other reasons?) or you just didn't have
the time to source out a dedicated RAID card?
Mostly timescales - it was a typical 'Director-driven' project where they havered around for ages and then suddenly wanted it yesterday. All the kit was already in stock and so...
-
Bryan J. Smith -
Chris Mauritz -
HaraldFinnås -
Nigel kendrick -
Peter Farrow -
Sudev Barar -
Tony