The recent thread on Anaconda and RAID10 made me start to think about how to partition a server I'm about to set up. I have two 146GB SCSI drives on an IBM x3550. It will be used as a build system. As such, there is no critical data on these systems, as the source code will be checked out of our source control system, and the build results are copied to another system. I usually build my systems with Kickstart, so if a disk dies, I can rebuild it quickly.
Given all that, how would you partition these disks? I keep going back and forth between various options (HW RAID, SW RAID, LVM, etc.). I guess speed is more important to me than redundancy. I'm tempted to install the OS on one drive and use the entire second drive for data. This way I can rebuild or upgrade the OS without touching the data. But that will waste a lot of disk space, as the OS does not need 146GB.
The only thing I'm pretty sure of is to put 2GB of swap on each drive, but after that everything is still in the air. I am looking for any and all suggestions from the collective wisdom and experience of this list.
Thanks, Alfred
Am Mittwoch, 9. Mai 2007 schrieb Alfred von Campe:
The recent thread on Anaconda and RAID10 made me start to think about how to partition a server I'm about to set up. I have two 146GB SCSI drives on an IBM x3550. It will be used as a build system. As such, there is no critical data on these systems, as the source code will be checked out of our source control system, and the build results are copied to another system. I usually build my systems with Kickstart, so if a disk dies, I can rebuild it quickly.
Given all that, how would you partition these disks? I keep going back and forth between various options (HW RAID, SW RAID, LVM, etc.). I guess speed is more important to me than redundancy. I'm tempted to install the OS on one drive and use the entire second drive for data. This way I can rebuild or upgrade the OS without touching the data. But that will waste a lot of disk space, as the OS does not need 146GB.
The only thing I'm pretty sure of is to put 2GB of swap on each drive, but after that everything is still in the air. I am looking for any and all suggestions from the collective wisdom and experience of this list.
Ask yourself this question: Does the company loose money when the build system is down for restore? How much? How long does a restore take?
Mirroring disks is not a replacement for backup. It is a way to improve availability of a system (no downtime when a disc dies), so it might even be interesting when there is no important data on the machine. If this is important for you use RAID-1 for the entire discs.
If decreased availability is not a problem for you (you can easily afford a day of downtime when a disc dies) use RAID-0 for the entire discs. It will give you a nice performance boost. Especially on a build host people will love the extra performance of the disc array.
A combination of RAID-0 and RAID-1 may also be an option: Make a small RAID-1 partition for the operating system (say 20GB) and a big RAID-0 partition for the data. This way you will get maximum performance on the data partition, but when a disc dies you do not need to reinstall the operating system. Just put in a new disc, let the RAID-1 rebuild itself in the background and restore your data. This can reduce the downtime (and the amount of work for you) when a disc dies considerably.
HW vs SW RAID: Kind of a religious question. HW has some advantages when using RAID-5 or RAID-6 (less CPU load). When using RAID-0 or RAID-1 there should not be any difference performance wise. HW RAID gives you some advantages in terms of handling, i.e. hotplugging of discs, nice administration console, RAID-10 during install ;-), etc. It's up to you to decide whether it is worth the money. Plus you need to find a controller that is well supported in Linux.
regards, Andreas Micklei
P.s. Putting lots of RAM into the machine (for the buffer cache) has more impact than RAID-0 in my experience. Of course that depends on your filesystem usage pattern.
P.p.s. Creating one swap partition on each disc is correct, because swapping to RAID-0 is useless. Only if you decide to use RAID-1 for the whole disc you should also swap to RAID-1.
HW vs SW RAID: Kind of a religious question. HW has some advantages when using RAID-5 or RAID-6 (less CPU load). When using RAID-0 or RAID-1 there should not be any difference performance wise. HW RAID gives you some advantages in terms of handling, i.e. hotplugging of discs, nice administration console, RAID-10 during install ;-), etc. It's up to you to decide whether it is worth the money. Plus you need to find a controller that is well supported in Linux.
Hardware raid that comes with bbu write caches normally have a speed boost in addition to the extra data safety too.
P.p.s. Creating one swap partition on each disc is correct, because swapping to RAID-0 is useless. Only if you decide to use RAID-1 for the whole disc you should also swap to RAID-1.
creating one swap partition on each disc is equivalent to swapping to raid 0...linux automatically strips/spreads swap data to all available swap areas.
Am Mittwoch, 9. Mai 2007 schrieb Feizhou:
HW vs SW RAID: Kind of a religious question. HW has some advantages when using RAID-5 or RAID-6 (less CPU load). When using RAID-0 or RAID-1 there should not be any difference performance wise. HW RAID gives you some advantages in terms of handling, i.e. hotplugging of discs, nice administration console, RAID-10 during install ;-), etc. It's up to you to decide whether it is worth the money. Plus you need to find a controller that is well supported in Linux.
Hardware raid that comes with bbu write caches normally have a speed boost in addition to the extra data safety too.
Also on RAID-0 and RAID-1? I have only used HW RAID without bbu, so I have no experience with that. However you can tweak your filesystem and disks to do write caching too, especially if you have plenty of RAM. Of course this means living dangerously, so when you need performance and data safety HW RAID with bbu is the only option of course.
P.p.s. Creating one swap partition on each disc is correct, because swapping to RAID-0 is useless. Only if you decide to use RAID-1 for the whole disc you should also swap to RAID-1.
creating one swap partition on each disc is equivalent to swapping to raid 0...linux automatically strips/spreads swap data to all available swap areas.
Exactly my point. According to the HOWTO swapping to RAID-0 even performs worse IIRC.
Andreas Micklei wrote:
Am Mittwoch, 9. Mai 2007 schrieb Feizhou:
HW vs SW RAID: Kind of a religious question. HW has some advantages when using RAID-5 or RAID-6 (less CPU load). When using RAID-0 or RAID-1 there should not be any difference performance wise. HW RAID gives you some advantages in terms of handling, i.e. hotplugging of discs, nice administration console, RAID-10 during install ;-), etc. It's up to you to decide whether it is worth the money. Plus you need to find a controller that is well supported in Linux.
Hardware raid that comes with bbu write caches normally have a speed boost in addition to the extra data safety too.
Also on RAID-0 and RAID-1? I have only used HW RAID without bbu, so I have no experience with that. However you can tweak your filesystem and disks to do write caching too, especially if you have plenty of RAM. Of course this means living dangerously, so when you need performance and data safety HW RAID with bbu is the only option of course.
Why did you think I specifically said hardware raid that comes with bbu memory caches? RAM will always be faster than disk so if the hardware raid stores stuff first in a bbu memory cache, it will always beat software raid even on raid0,raid1.
Hardware raid without a bbu memory cache will always be slower on raid5 even if the raid processor is beefy enough.
Am Donnerstag, 10. Mai 2007 schrieb Feizhou:
Andreas Micklei wrote:
Am Mittwoch, 9. Mai 2007 schrieb Feizhou:
HW vs SW RAID: Kind of a religious question. HW has some advantages when using RAID-5 or RAID-6 (less CPU load). When using RAID-0 or RAID-1 there should not be any difference performance wise. HW RAID gives you some advantages in terms of handling, i.e. hotplugging of discs, nice administration console, RAID-10 during install ;-), etc. It's up to you to decide whether it is worth the money. Plus you need to find a controller that is well supported in Linux.
Hardware raid that comes with bbu write caches normally have a speed boost in addition to the extra data safety too.
Also on RAID-0 and RAID-1? I have only used HW RAID without bbu, so I have no experience with that. However you can tweak your filesystem and disks to do write caching too, especially if you have plenty of RAM. Of course this means living dangerously, so when you need performance and data safety HW RAID with bbu is the only option of course.
Why did you think I specifically said hardware raid that comes with bbu memory caches?
Because you said "Hardware raid that comes with bbu write caches normally have a speed boost in addition to the extra data safety too."? ;-)
RAM will always be faster than disk so if the hardware raid stores stuff first in a bbu memory cache, it will always beat software raid even on raid0,raid1.
Does'nt Linux have a memory cache on it's own? Putting lot's of RAM into a machine, using a fast filesystem, i.e. XFS and tuning it to not write everything to disc immideately should give equal performance. It should be even better, because you can put much more RAM into a machine than on the RAID controller.
Of course this does not give you the same level of data safety. Far from it!
Hardware raid without a bbu memory cache will always be slower on raid5 even if the raid processor is beefy enough.
I use HW RAID with a memory cache but without bbu which gives the same level of performance. Of course this is dangerous and might eat the data when the machine dies. However with redundant power supply and UPS this is not so likely (and yes, we have backups).
Also on RAID-0 and RAID-1? I have only used HW RAID without bbu, so I have no experience with that. However you can tweak your filesystem and disks to do write caching too, especially if you have plenty of RAM. Of course this means living dangerously, so when you need performance and data safety HW RAID with bbu is the only option of course.
Why did you think I specifically said hardware raid that comes with bbu memory caches?
Because you said "Hardware raid that comes with bbu write caches normally have a speed boost in addition to the extra data safety too."? ;-)
RAID-0/RAID-1 with a filesystem using lots of cache in memory will not be faster in every scenario. Applications that call fsync will bypass the memory cache and thus be reduced to disk speed rather than memory speed in the case of hardware raid + bbu memory cache.
RAM will always be faster than disk so if the hardware raid stores stuff first in a bbu memory cache, it will always beat software raid even on raid0,raid1.
Does'nt Linux have a memory cache on it's own? Putting lot's of RAM into a machine, using a fast filesystem, i.e. XFS and tuning it to not write everything to disc immideately should give equal performance. It should be even better, because you can put much more RAM into a machine than on the RAID controller.
People do not turn off fsync. They may mount async but that does not affect fsync calls from databases, mtas, and other data important applications.
Of course this does not give you the same level of data safety. Far from it!
Another way could to get a NVM card or BBU RAM drive for the journal of filesystems that support full data journalling but the price difference makes getting a RAID card a no brainer in most cases.
Hardware raid without a bbu memory cache will always be slower on raid5 even if the raid processor is beefy enough.
I use HW RAID with a memory cache but without bbu which gives the same level of performance. Of course this is dangerous and might eat the data when the machine dies. However with redundant power supply and UPS this is not so likely (and yes, we have backups).
Yes, so long as you have a large enough memory cache, hardware raid 5 will be faster than software raid.
Am Donnerstag, 10. Mai 2007 schrieb Feizhou:
People do not turn off fsync. They may mount async but that does not affect fsync calls from databases, mtas, and other data important applications.
Tanks! Interesting to keep in mind.
Alfred von Campe wrote:
The recent thread on Anaconda and RAID10 made me start to think about how to partition a server I'm about to set up. I have two 146GB SCSI drives on an IBM x3550. It will be used as a build system. As such, there is no critical data on these systems, as the source code will be checked out of our source control system, and the build results are copied to another system. I usually build my systems with Kickstart, so if a disk dies, I can rebuild it quickly.
Given all that, how would you partition these disks? I keep going back and forth between various options (HW RAID, SW RAID, LVM, etc.). I guess speed is more important to me than redundancy. I'm tempted to install the OS on one drive and use the entire second drive for data. This way I can rebuild or upgrade the OS without touching the data. But that will waste a lot of disk space, as the OS does not need 146GB.
The only thing I'm pretty sure of is to put 2GB of swap on each drive, but after that everything is still in the air. I am looking for any and all suggestions from the collective wisdom and experience of this list.
Three raid1 sets:
raid1 #1 = / raid1 #2 = swap raid1 #3 = rest of disk on /home
for the simple fact that a dead disk won't bring down your system and halt your builds until your rebuild the machine. But if you really only care about max speed and are not worried about crashes & their consequences, then replace the raid1 with raid0.
I have no reason for using LVM on boot/OS/system partitions. If I have something that fills the disk that much, I move it to an other storage device. In your case, striped LVM could be used instead of raid0.
for the simple fact that a dead disk won't bring down your system and halt your builds until your rebuild the machine. But if you really only care about max speed and are not worried about crashes & their consequences, then replace the raid1 with raid0.
just remember that grub does NOT support raid0. you have to put / or at a minimum /boot on a raid1 device
Alfred von Campe wrote:
The recent thread on Anaconda and RAID10 made me start to think about how to partition a server I'm about to set up. I have two 146GB SCSI drives on an IBM x3550. It will be used as a build system. As such, there is no critical data on these systems, as the source code will be checked out of our source control system, and the build results are copied to another system. I usually build my systems with Kickstart, so if a disk dies, I can rebuild it quickly.
I used kickstart, pxe, tftp and dhcp to manage a cluster of mail servers. Two disk 1U boxes too. /, swap, /var (logging was to a central log host).
Given all that, how would you partition these disks? I keep going back and forth between various options (HW RAID, SW RAID, LVM, etc.). I guess speed is more important to me than redundancy. I'm tempted to install the OS on one drive and use the entire second drive for data. This way I can rebuild or upgrade the OS without touching the data. But that will waste a lot of disk space, as the OS does not need 146GB.
A scsi raid controller with write cache? Hardware raid definitely. Especially if the driver supports the write cache.
The only thing I'm pretty sure of is to put 2GB of swap on each drive, but after that everything is still in the air. I am looking for any and all suggestions from the collective wisdom and experience of this list.
swap should go on a raid1 device whether partition or swap file.
-
Alfred von Campe -
Andreas Micklei -
Feizhou -
Toby Bluhm