Hi there,
I am currently setting up a server that will house my backups (simply using rsync).
This system has 4 X 500 gb drives and I am looking to raid for max drive space and data safety. Performance is not so much a concern.
My experience with software raids in nil, so some of these may seem like pretty dumb questions.
I was thinking a raid 1 is probably sufficient.
Would it be best to raid 1 two drives each and LVM them together?
My next question would be about how to do this as I have never done a linux software raid.
D
For controller, what is the interface on your drives?? SCSI, SAS??
John Plemons
dnk wrote:
Hi there,
I am currently setting up a server that will house my backups (simply using rsync).
This system has 4 X 500 gb drives and I am looking to raid for max drive space and data safety. Performance is not so much a concern.
My experience with software raids in nil, so some of these may seem like pretty dumb questions.
I was thinking a raid 1 is probably sufficient.
Would it be best to raid 1 two drives each and LVM them together?
My next question would be about how to do this as I have never done a linux software raid.
D
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
For controller, what is the interface on your drives?? SCSI, SAS??
Dell 2950, SAS 6 Host Bus Controller.
Integrated SAS 6/i(base): 4 port SAS controller (does support RAID 0/1) But I don't know if that is descent hw raid or crap raid...
JD
On 18-Feb-09, at 2:01 AM, John Doe wrote:
For controller, what is the interface on your drives?? SCSI, SAS??
Dell 2950, SAS 6 Host Bus Controller.
Integrated SAS 6/i(base): 4 port SAS controller (does support RAID 0/1) But I don't know if that is descent hw raid or crap raid...
JD
This was kind of the reason I was thinking software raid.
Has anyone had any raid experience with this card?
d
On Wed, Feb 18, 2009 at 08:13:23AM -0800, dnk wrote:
On 18-Feb-09, at 2:01 AM, John Doe wrote:
For controller, what is the interface on your drives?? SCSI, SAS??
Dell 2950, SAS 6 Host Bus Controller.
Integrated SAS 6/i(base): 4 port SAS controller (does support RAID 0/1) But I don't know if that is descent hw raid or crap raid...
JD
This was kind of the reason I was thinking software raid.
Has anyone had any raid experience with this card?
d
So this isn't the PERC then? The PERC should be real hardware RAID...
Ray
On Wed, 2009-02-18 at 08:13 -0800, dnk wrote:
On 18-Feb-09, at 2:01 AM, John Doe wrote:
For controller, what is the interface on your drives?? SCSI, SAS??
Dell 2950, SAS 6 Host Bus Controller.
Integrated SAS 6/i(base): 4 port SAS controller (does support RAID 0/1) But I don't know if that is descent hw raid or crap raid...
JD
This was kind of the reason I was thinking software raid.
Has anyone had any raid experience with this card?
Yep - it's real HW raid, though with a 2950, I would have gone for the PERC instead. The integrated SAS is usually a Fusion-MPT (LSI/Symbios) card. Do a 'lspci' to be sure. I've got a few of these configured and I can check the raid status with the mpt-status command (from the mpt-status rpm). Note that doing a 'fdisk -l' only yields one disk when RAID is setup. So yes - it's real HW raid. Just not much in the way of cache, which is why I prefer using PERCs instead...
-I
On Tue, Feb 17, 2009 at 01:52:52PM -0800, dnk wrote:
Hi there,
I am currently setting up a server that will house my backups (simply using rsync).
This system has 4 X 500 gb drives and I am looking to raid for max drive space and data safety. Performance is not so much a concern.
My experience with software raids in nil, so some of these may seem like pretty dumb questions.
I was thinking a raid 1 is probably sufficient.
Would it be best to raid 1 two drives each and LVM them together?
This probably gives you both the best reliability and the best performance (RAID 10). You can potentially lose two drives (although they'd need to be on opposite mirrored subsets) and be OK.
If you want to maximize space, RAID5 would work as well. You'd get nearly 1500GB out of this setup but could only survive one disk failure.
Your other option would be RAID6 which would net you the same amount of space as RAID10 with four drives and also allow for two drive failures (any of the drives in this case). If you anticipate adding additional drives later, this would allow you to add one drive at a time instead of requiring a pair.
RAID6 has a write performance penalty, but I imagine it would still run well enough to handle backups.
I think I might lean towards the RAID6 option in this case.... same space as RAID10, but allows you to lose *any* two drives and still maintain functionality at the expense of some write performance.
My next question would be about how to do this as I have never done a linux software raid.
D
Ray
dnk wrote:
Hi there,
I am currently setting up a server that will house my backups (simply using rsync).
This system has 4 X 500 gb drives and I am looking to raid for max drive space and data safety. Performance is not so much a concern.
My experience with software raids in nil, so some of these may seem like pretty dumb questions.
I was thinking a raid 1 is probably sufficient.
Would it be best to raid 1 two drives each and LVM them together?
My next question would be about how to do this as I have never done a linux software raid.
D
Linux software raid works fine and I use this recipe http://linux-raid.osdl.org/index.php/Partitioning_RAID_/_LVM_on_RAID If you can afford a proper hardware raid controller for raid 6, that would be of better performance than linux software raid, other than that I have found performance of linux based md using raid 1 to work fine.
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On 17-Feb-09, at 2:08 PM, Rob Kampen wrote:
Linux software raid works fine and I use this recipe http://linux-raid.osdl.org/index.php/Partitioning_RAID_/_LVM_on_RAID If you can afford a proper hardware raid controller for raid 6, that would be of better performance than linux software raid, other than that I have found performance of linux based md using raid 1 to work fine.
Rob,
So are you using 4 drives, and making two different raid 1 (2 drives each), then joining them via LVM?
d
Dnk, I use two drives with linux raid 1 sets for the OS for all my CentOS machines, drives are cheaper than my rebuild time and hassle. I actually use 3 partitions mirrored on each: /boot of 100M; swap of 2 time RAM (disk is cheap); 70G as /, then the remainder is extended partition for lvm - in this case the extended partition is not mirrored as the contents are for mythTV recordings that I don't mind loosing. I have a HD TV card that takes 5G per hour of recording so space matters more than redundancy - also use xfs on this lvm part as it seems to work better for huge files. In your case I would set up both the raid 1 sets, put lvm volumes in place and allocate as needed for the mount points you want to have. Because you are using lvm, if you get the allocation wrong it can be remedied without too much hassle. Hope this helps. Rob
dnk wrote:
On 17-Feb-09, at 2:08 PM, Rob Kampen wrote:
Linux software raid works fine and I use this recipe http://linux-raid.osdl.org/index.php/Partitioning_RAID_/_LVM_on_RAID If you can afford a proper hardware raid controller for raid 6, that would be of better performance than linux software raid, other than that I have found performance of linux based md using raid 1 to work fine.
Rob,
So are you using 4 drives, and making two different raid 1 (2 drives each), then joining them via LVM?
d
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Would it be best to raid 1 two drives each and LVM them together?
My next question would be about how to do this as I have never done a linux software raid.
I would do it this way if they are not system disks:
Eg: sdc + sdd = md0 (raid 1) sde + sdf = md1 (raid 1) md0 + md1 = md2 (raid 0)
Use md2 as a physical volume
on 2-17-2009 1:52 PM dnk spake the following:
Hi there,
I am currently setting up a server that will house my backups (simply using rsync).
This system has 4 X 500 gb drives and I am looking to raid for max drive space and data safety. Performance is not so much a concern.
Max size would be raid 5, max safety would be raid 10 or lvm over raid 1's
My experience with software raids in nil, so some of these may seem like pretty dumb questions.
I was thinking a raid 1 is probably sufficient.
Would it be best to raid 1 two drives each and LVM them together?
My next question would be about how to do this as I have never done a linux software raid.
http://tldp.org/HOWTO/Software-RAID-HOWTO.html
http://www.linuxdevcenter.com/pub/a/linux/2002/12/05/RAID.html
D
On 18-Feb-09, at 9:14 AM, Scott Silva wrote:
http://tldp.org/HOWTO/Software-RAID-HOWTO.html
http://www.linuxdevcenter.com/pub/a/linux/2002/12/05/RAID.html
If i am to understand the tutorials right, does one create the raid/ lvm after install? Or do you boot off the disk, use these tools, then install on existing structures. I was trying to do it through the disk druid during install.
d
on 2-18-2009 10:13 AM dnk spake the following:
On 18-Feb-09, at 9:14 AM, Scott Silva wrote:
http://tldp.org/HOWTO/Software-RAID-HOWTO.html
http://www.linuxdevcenter.com/pub/a/linux/2002/12/05/RAID.html
If i am to understand the tutorials right, does one create the raid/lvm after install? Or do you boot off the disk, use these tools, then install on existing structures. I was trying to do it through the disk druid during install.
d
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
You can make LVM over raid 1's in Disk Druid, but I don't think it will do raid 10. And you cannot boot from software raid 5 (yet).
Scott Silva wrote:
You can make LVM over raid 1's in Disk Druid, but I don't think it will do raid 10. And you cannot boot from software raid 5 (yet).
a LVM over several raid 1's is effectively raid10 as LVM will stripe the volumes across the devices. It would be nice if LVM could do mirorring too (like LVM on AIX does) and was tighter integrated with the file system tools (again, like LVM on AIX... grow a LV and it grows the JFS thats sitting on it, transparently and online).
on 2-18-2009 11:12 AM John R Pierce spake the following:
Scott Silva wrote:
You can make LVM over raid 1's in Disk Druid, but I don't think it will do raid 10. And you cannot boot from software raid 5 (yet).
a LVM over several raid 1's is effectively raid10 as LVM will stripe the volumes across the devices. It would be nice if LVM could do mirorring too (like LVM on AIX does) and was tighter integrated with the file system tools (again, like LVM on AIX... grow a LV and it grows the JFS thats sitting on it, transparently and online).
Effectively and actually are two different things. Some people seem to be very shy of LVM, even though it seems to be fairly mature technology. I didn't think the LVM will stripe over the raid 1's by itself, I thought you had to set stripes according to the underlying raid structure to get optimum speed.
On Wed, Feb 18, 2009 at 11:12:13AM -0800, John R Pierce wrote:
Scott Silva wrote:
You can make LVM over raid 1's in Disk Druid, but I don't think it will do raid 10. And you cannot boot from software raid 5 (yet).
a LVM over several raid 1's is effectively raid10 as LVM will stripe the volumes across the devices. It would be nice if LVM could do mirorring too (like LVM on AIX does) and was tighter integrated with the file system tools (again, like LVM on AIX... grow a LV and it grows the JFS thats sitting on it, transparently and online).
Can't Linux LVM do mirroring? I swear I read that it could in the man page. Never have tried it however and you certainly can't set it up from disk druid in anaconda.
Ray
Ray Van Dolson wrote:
Can't Linux LVM do mirroring? I swear I read that it could in the man page. Never have tried it however and you certainly can't set it up from disk druid in anaconda.
dunno. the word 'mirror occurs exactly once in the man page for lvm(8)
lvconvert -- Convert a logical volume from linear to mirror or snapshot.
the man page on lvconvert has more info, and this http://www.centos.org/docs/5/html/5.2/Cluster_Logical_Volume_Manager/mirrore... certainly implies CentOS5 supports it.
glad to know thats finally there. however, it doesn't seem to support little things like hot spares or automatic rebuilding. in fact, the rebuild process looks involved and convoluted... http://www.centos.org/docs/5/html/5.2/Cluster_Logical_Volume_Manager/mirrorr...
On Wed, Feb 18, 2009 at 11:51:55AM -0800, John R Pierce wrote:
Ray Van Dolson wrote:
Can't Linux LVM do mirroring? I swear I read that it could in the man page. Never have tried it however and you certainly can't set it up from disk druid in anaconda.
dunno. the word 'mirror occurs exactly once in the man page for lvm(8)
lvconvert -- Convert a logical volume from linear to mirror orsnapshot.
the man page on lvconvert has more info, and this http://www.centos.org/docs/5/html/5.2/Cluster_Logical_Volume_Manager/mirrore... certainly implies CentOS5 supports it.
glad to know thats finally there. however, it doesn't seem to support little things like hot spares or automatic rebuilding. in fact, the rebuild process looks involved and convoluted... http://www.centos.org/docs/5/html/5.2/Cluster_Logical_Volume_Manager/mirrorr...
The other side of the coin (as I think you mentioned) is that many are not comfortable having LVM handle the mirroring. Are its mirroring abilities as mature or fast as md? It's certainly not documented as well at the very least. :)
For now, I prefer doing my mirroring with md and the striping with LVM. Would be interesting to hear anyone else's experiences with LVM mirroring however.
Ray
On Thu, Feb 19, 2009 at 4:22 AM, Ray Van Dolson rayvd@bludgeon.org wrote:
The other side of the coin (as I think you mentioned) is that many are not comfortable having LVM handle the mirroring. Are its mirroring abilities as mature or fast as md? It's certainly not documented as well at the very least. :)
I remember googling for this before setting up a server some weeks ago and somebody did a benchmark. The general conclusion was stick to md for RAID 1, it has better performance. IIRC, one of the reason was while md1 will read from both disk, LVM mirror apparently only reads from the "master" unless it fails.
Furthermore, given the nightmare of a time I'm having trying to restore a LVM PV sitting across 3 pairs of md RAID 1, I'll strongly recommend against tempting fate by using LVM for mirroring as well.
Thankfully for the underlying md mirror, I can at least activate the LVM vg and offload data in rescue mode even if it won't work off a normal boot.
Noob Centos Admin schrieb:
On Thu, Feb 19, 2009 at 4:22 AM, Ray Van Dolson <rayvd@bludgeon.org mailto:rayvd@bludgeon.org> wrote:
The other side of the coin (as I think you mentioned) is that many are not comfortable having LVM handle the mirroring. Are its mirroring abilities as mature or fast as md? It's certainly not documented as well at the very least. :)I remember googling for this before setting up a server some weeks ago and somebody did a benchmark. The general conclusion was stick to md for RAID 1, it has better performance. IIRC, one of the reason was while md1 will read from both disk, LVM mirror apparently only reads from the "master" unless it fails.
"md1 will read from both disk" is not true in general. RAID1 md reads from one disk only; it uses the other one in case the first one fails. No performance gain from multiple copies.
You can easily see this for yourself by setting up a RAID1 from e.g. sda1 and sdb1 - /proc/mdstat is:
Personalities : [raid1] md1 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU]
and then comparing the output of hdparm -tT :
/dev/sda1: Timing cached reads: 29368 MB in 2.00 seconds = 14711.93 MB/sec Timing buffered disk reads: 100 MB in 0.92 seconds = 108.79 MB/sec
/dev/md1: Timing cached reads: 28000 MB in 2.00 seconds = 14023.66 MB/sec Timing buffered disk reads: 100 MB in 0.95 seconds = 105.81 MB/sec
/dev/sdb1: Timing cached reads: 23780 MB in 2.00 seconds = 11907.30 MB/sec Timing buffered disk reads: 100 MB in 0.98 seconds = 102.51 MB/sec
To get performance gain in RAID1 mode you need hardware RAID1.
HTH,
Kay
"md1 will read from both disk" is not true in general. RAID1 md reads from one disk only; it uses the other one in case the first one fails. No performance gain from multiple copies.
I beg to differ. I have disks in a raid1 md array and iostat -x 1 will show reads coming off both disks. Unless you do not have the multipath module loaded, md will read off both disks. Now whether md will read equally off both disks, that certainly will not be true in general.
You can easily see this for yourself by setting up a RAID1 from e.g. sda1 and sdb1 - /proc/mdstat is:
Personalities : [raid1] md1 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU]
and then comparing the output of hdparm -tT :
ROTFL.
How about using the proper tool (iostat) and generating some disk load instead?
To get performance gain in RAID1 mode you need hardware RAID1.
Bollocks. The only area in which hardware raid has a significant performance advantage over software raid is raid5/6 given sufficient cache memory and processing power.
On Fri, 2009-02-20 at 22:52 +0800, Chan Chung Hang Christopher wrote:
Bollocks. The only area in which hardware raid has a significant performance advantage over software raid is raid5/6 given sufficient cache memory and processing power.
I'd have to say no on the processing power for RAID 5. Moore's law has grown CPU capabilities over the last 15 or so years. HW RAID controllers haven't gotten that much faster because they haven't needed to. It's faster to do it in software, though it's preferable to offload it to HW RAID so that any apps aren't affected directly.
I would agree on that cache memory is an advantage, especially when considering battery-backed cache memory.
But those aren't the only significant areas. HW RAID allows for hot-swap and pain-free (meaning zero commands needed) disk replacement.
-I
Ian Forde wrote:
On Fri, 2009-02-20 at 22:52 +0800, Chan Chung Hang Christopher wrote:
Bollocks. The only area in which hardware raid has a significant performance advantage over software raid is raid5/6 given sufficient cache memory and processing power.
I'd have to say no on the processing power for RAID 5. Moore's law has grown CPU capabilities over the last 15 or so years. HW RAID controllers haven't gotten that much faster because they haven't needed to. It's faster to do it in software, though it's preferable to offload it to HW RAID so that any apps aren't affected directly.
You will have to prove that. I have previously posted posts with links to benchmarks that show that hardware raid with sufficient processing power beat the pants of software raid when it comes to raid5/6 implementations. Hardware raid cards no longer come with crappy i960 cpus.
I would agree on that cache memory is an advantage, especially when considering battery-backed cache memory.
There is more to it. That cache memory also cuts down on bus traffic but the real kicker is that there is no bus contention between the board's cpu and disk data whereas software raid needs to read of the disks for its calculations and therefore suffers latencies that hardware raid boards (which have direct connections to disks) do not. Of course, if the cache size is insufficient, then the hardware raid board will not perform much better if not worse than software raid.
But those aren't the only significant areas. HW RAID allows for hot-swap and pain-free (meaning zero commands needed) disk replacement.
Hmm...really? I guess it depends on the board. (okay, okay, thinking of antique 3ware 750x series may not be fair)
On Sat, 2009-02-21 at 08:40 +0800, Chan Chung Hang Christopher wrote:
Ian Forde wrote:
I'd have to say no on the processing power for RAID 5. Moore's law has grown CPU capabilities over the last 15 or so years. HW RAID controllers haven't gotten that much faster because they haven't needed to. It's faster to do it in software, though it's preferable to offload it to HW RAID so that any apps aren't affected directly.
You will have to prove that. I have previously posted posts with links to benchmarks that show that hardware raid with sufficient processing power beat the pants of software raid when it comes to raid5/6 implementations. Hardware raid cards no longer come with crappy i960 cpus.
Just by doing some quick googling, I came across:
http://blogs.zdnet.com/storage/?p=126 http://storagemojo.com/2007/04/24/mo-better-zfs-performance-stats/ http://milek.blogspot.com/2007/04/hw-raid-vs-zfs-software-raid-part-iii.html
Now, bear in mind that I'm no ZFS fanboy, but I'm saying that it's not so cut and dry anymore. The equation changes, of course, when we're talking about a purposed fileserver versus an application server that needs RAID. (The app server can suffer because its losing access to CPU resources.) But the point of contention is still there. Both are viable solutions, when considering that SW RAID was never a serious contender for performance over the years, look at where it is now. This tells me that it's trending up towards equaling or bettering HW RAID performance. And that's not talking about price points. When throwing that in...
But again - I still like HW RAID. I think we're in agreement on this.
I would agree on that cache memory is an advantage, especially when considering battery-backed cache memory.
There is more to it. That cache memory also cuts down on bus traffic but the real kicker is that there is no bus contention between the board's cpu and disk data whereas software raid needs to read of the disks for its calculations and therefore suffers latencies that hardware raid boards (which have direct connections to disks) do not. Of course, if the cache size is insufficient, then the hardware raid board will not perform much better if not worse than software raid.
Indeed.
But those aren't the only significant areas. HW RAID allows for hot-swap and pain-free (meaning zero commands needed) disk replacement.
Hmm...really? I guess it depends on the board. (okay, okay, thinking of antique 3ware 750x series may not be fair)
I was thinking about when I was running a farm of 500 HP DL-x80 series boxes and disk replacement became a 9x5 job that we farmed out. Just give a list of servers and locations (first drive or second drive) and the person could pull old drives out, put new drives in, and resync was automatic. Same thing is true for Dell PERC hardware. I note that that's not necessarily true with ALL HW RAID controllers, as they have to support hot-swap, and the chassis has to have hot-swap slots. But still, I've only seen one SW RAID implementation that does auto-sync. That's the Infrant ReadyNAS (http://www.readynas.com). I wonder how they did it? Might not be a bad idea to see how they're able to use mdadm to detect and autosync drives. I don't *ever* want to go through something like:
http://kev.coolcavemen.com/2008/07/heroic-journey-to-raid-5-data-recovery/
Not when a little planning can help me skip it... ;)
-I
Ian Forde wrote:
Might not be a bad idea to see how they're able to use mdadm to detect and autosync drives. I don't *ever* want to go through something like:
http://kev.coolcavemen.com/2008/07/heroic-journey-to-raid-5-data-recovery/
Not when a little planning can help me skip it... ;)
If you are really concerned about data recovery and can chunk up your filesystem mount points so things fit on a single disk (usually not too hard with 1 or 1.5 TB drives available now) just use software raid1 since you can simply mount any single disk from it and access the files. It becomes much more difficult with other raid levels or multi-disk lvm.
On Sat, 2009-02-21 at 17:24 -0600, Les Mikesell wrote:
Ian Forde wrote:
Might not be a bad idea to see how they're able to use mdadm to detect and autosync drives. I don't *ever* want to go through something like:
http://kev.coolcavemen.com/2008/07/heroic-journey-to-raid-5-data-recovery/
Not when a little planning can help me skip it... ;)
If you are really concerned about data recovery and can chunk up your filesystem mount points so things fit on a single disk (usually not too hard with 1 or 1.5 TB drives available now) just use software raid1 since you can simply mount any single disk from it and access the files. It becomes much more difficult with other raid levels or multi-disk lvm.
My point is that at home, I'd rather do network mounts to a fileserver utilizing HW RAID. At work, I'd rather use HW RAID with hot-swap disks. This way, there's are no hoops to go through. Time is a more important resource to me... SW RAID is a path that I went down well over a decade ago in Solaris (DiskSuite and Veritas VM), followed by Linux mdadm. If you've ever had to do a Veritas encapsulated boot disk recovery, you'll know why I'd rather never go down that road *ever again*... ;)
-I
-I
Ian Forde wrote:
On Sat, 2009-02-21 at 17:24 -0600, Les Mikesell wrote:
Ian Forde wrote:
Might not be a bad idea to see how they're able to use mdadm to detect and autosync drives. I don't *ever* want to go through something like:
http://kev.coolcavemen.com/2008/07/heroic-journey-to-raid-5-data-recovery/
Not when a little planning can help me skip it... ;)
If you are really concerned about data recovery and can chunk up your filesystem mount points so things fit on a single disk (usually not too hard with 1 or 1.5 TB drives available now) just use software raid1 since you can simply mount any single disk from it and access the files. It becomes much more difficult with other raid levels or multi-disk lvm.
My point is that at home, I'd rather do network mounts to a fileserver utilizing HW RAID. At work, I'd rather use HW RAID with hot-swap disks. This way, there's are no hoops to go through. Time is a more important resource to me... SW RAID is a path that I went down well over a decade ago in Solaris (DiskSuite and Veritas VM), followed by Linux mdadm. If you've ever had to do a Veritas encapsulated boot disk recovery, you'll know why I'd rather never go down that road *ever again*... ;)
Yes, but raid1 in software has none of those problems, since as far as the boot loader is concerned, you are booting from a single drive. And there is a trade-off in complexity, since sw raid works the same on Linux across different hardware and you need to round up different vendors instructions and utilities for hardware raid - and have a backup controller around for recovery.
On Sat, 2009-02-21 at 18:09 -0600, Les Mikesell wrote:
Yes, but raid1 in software has none of those problems, since as far as the boot loader is concerned, you are booting from a single drive. And there is a trade-off in complexity, since sw raid works the same on Linux across different hardware and you need to round up different vendors instructions and utilities for hardware raid - and have a backup controller around for recovery.
RAID in software, whether RAID1 or RAID5/6, always has manual steps involved in recovery. If one is using standardized hardware, such as HP DL-x80 hardware or Dell x950 boxes, HW RAID obviates the need for a "recovery procedure". It's just easier. You can still boot from a single drive, since that's what the bootloader sees. There are no vendor instructions or utilities needed for recovery. Nor is there a backup controller needed. The *only* time I'd use software RAID on Linux is if I didn't have a standard hardware base that supported hotswap and commandless recovery, which in any enterprise within which I were to be employed, I'd insist upon (and deploy)...
-I
Ian Forde wrote:
On Sat, 2009-02-21 at 18:09 -0600, Les Mikesell wrote:
Yes, but raid1 in software has none of those problems, since as far as the boot loader is concerned, you are booting from a single drive. And there is a trade-off in complexity, since sw raid works the same on Linux across different hardware and you need to round up different vendors instructions and utilities for hardware raid - and have a backup controller around for recovery.
RAID in software, whether RAID1 or RAID5/6, always has manual steps involved in recovery.
Don't forget that 'recovery' sometimes means taking the still-working drives and moving them to a new chassis.
If one is using standardized hardware, such as HP DL-x80 hardware or Dell x950 boxes, HW RAID obviates the need for a "recovery procedure".
As long as you have an exactly-matching chassis/motherboard/controller to move to.
It's just easier. You can still boot from a single drive, since that's what the bootloader sees. There are no vendor instructions or utilities needed for recovery. Nor is there a backup controller needed.
Everything breaks eventually. If yours hasn't yet, good luck with that.
The *only* time I'd use software RAID on Linux is if I didn't have a standard hardware base that supported hotswap and commandless recovery, which in any enterprise within which I were to be employed, I'd insist upon (and deploy)...
You can have hot spares in software raid if you can't be bothered to type 'mdadm --add ....' once every few years.
On Sun, Feb 22, 2009 at 7:05 PM, Ian Forde ian@duckland.org wrote:
RAID in software, whether RAID1 or RAID5/6, always has manual steps involved in recovery. If one is using standardized hardware, such as HP DL-x80 hardware or Dell x950 boxes, HW RAID obviates the need for a "recovery procedure". It's just easier. You can still boot from a single drive, since that's what the bootloader sees. There are no vendor instructions or utilities needed for recovery. Nor is there a backup controller needed.
If I have to do hardware raid, I'll definitely spec in a backup controller. Learnt this the hard way when my raid 5 controller died years after I first got it and I could no longer find a replacement.
For high budget projects, having the extra raid controller as insurance isn't a big deal. But for most budget setup and cost conscious clients, soft raid obviates that hardware dependency.
If I have to do hardware raid, I'll definitely spec in a backup controller. Learnt this the hard way when my raid 5 controller died years after I first got it and I could no longer find a replacement.
For high budget projects, having the extra raid controller as insurance isn't a big deal. But for most budget setup and cost conscious clients, soft raid obviates that hardware dependency.
You forgot flaky hardware. Boards that lose their configuration from time to time and what not...I heard stories about Mylex for example...
You will have to prove that. I have previously posted posts with links to benchmarks that show that hardware raid with sufficient processing power beat the pants of software raid when it comes to raid5/6 implementations. Hardware raid cards no longer come with crappy i960 cpus.
Just by doing some quick googling, I came across:
http://blogs.zdnet.com/storage/?p=126 http://storagemojo.com/2007/04/24/mo-better-zfs-performance-stats/ http://milek.blogspot.com/2007/04/hw-raid-vs-zfs-software-raid-part-iii.html
I have an issue with that benchmark. I don't know what that EMC is made of nevermind the fancy FC.
Regarding my posts...they were on the ubuntu-user list...not here...so...here are the links:
Thread post: https://lists.ubuntu.com/archives/edubuntu-users/2008-December/004887.html
Don't use underpowered hardward raid boards for raid5/6 http://www.chemistry.wustl.edu/~gelb/castle_raid.html
Use hardware raid boards with enough cpu/cache for raid5/6 http://www.linux.com/feature/140734
Chan Chung Hang Christopher schrieb:
"md1 will read from both disk" is not true in general. RAID1 md reads from one disk only; it uses the other one in case the first one fails. No performance gain from multiple copies.
I beg to differ. I have disks in a raid1 md array and iostat -x 1 will show reads coming off both disks. Unless you do not have the multipath
look more carefully - with the current 2.6.18-9.1.22 kernel the bulk of the data are read from one of the disks
module loaded, md will read off both disks. Now whether md will read equally off both disks, that certainly will not be true in general.
You can easily see this for yourself by setting up a RAID1 from e.g. sda1 and sdb1 - /proc/mdstat is:
Personalities : [raid1] md1 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU]
and then comparing the output of hdparm -tT :
ROTFL.
How about using the proper tool (iostat) and generating some disk load instead?
hdparm -tT tests one type of disk access, other tools test other aspects. I gave the hdparm numbers because everyone can reproduce them. For RAID0 with two disks you do see - using e.g. hdparm - the doubling of performance from two disks. If you take the time to read (or do) RAID benchmarks you'll discover that Linux software RAID1 is about as fast as a single disk (and RAID0 with two disks is about twice the speed). It's as simple as that.
To get performance gain in RAID1 mode you need hardware RAID1.
Bollocks. The only area in which hardware raid has a significant performance advantage over software raid is raid5/6 given sufficient cache memory and processing power.
We were talking about RAID1; RAID5/6 is a different area. Linux software RAID1 is a safeguard against disk failure; it's not designed for speed increase. There is a number of things that could be improved in Linux software RAID; read performance of RAID1 is one of them - this _is_ why some hardware RAID1 adapters indeed are faster than software. Read http://kernelnewbies.org/KernelProjects/Raid1ReadBalancing - since the 2.6.25 kernel a simple alternating read is implemented, but that does not take the access pattern into account.
So Linux software RAID1 is just mirroring - and it's good at that.
HTH,
Kay
Kay Diederichs wrote:
hdparm -tT tests one type of disk access, other tools test other aspects. I gave the hdparm numbers because everyone can reproduce them. For RAID0 with two disks you do see - using e.g. hdparm - the doubling of performance from two disks. If you take the time to read (or do) RAID benchmarks you'll discover that Linux software RAID1 is about as fast as a single disk (and RAID0 with two disks is about twice the speed). It's as simple as that.
maybe with a simple single threaded application. if there are concurrent read requests pending it will dispatch them to both drives.
On Sat, Feb 21, 2009 at 6:04 PM, John R Pierce pierce@hogranch.com wrote:
Kay Diederichs wrote:
hdparm -tT tests one type of disk access, other tools test other aspects. I gave the hdparm numbers because everyone can reproduce them. For RAID0 with two disks you do see - using e.g. hdparm - the doubling of performance from two disks. If you take the time to read (or do) RAID benchmarks you'll discover that Linux software RAID1 is about as fast as a single disk (and RAID0 with two disks is about twice the speed). It's as simple as that.
maybe with a simple single threaded application. if there are concurrent read requests pending it will dispatch them to both drives.
I'm waiting for a 10 hour backup to be completed before doing recovery on a server (ok recovery is a nice way to put it, truth is I gave up any hope of making the screwed LVM setup work and going to wipe/reinstall after the backup), I'll probably be able to try some tests.
However, I don't know enough to do this properly. So some questions:
Would running two CP command to copy 2 different set of files to two different targets suffice as a basic two thread test?
Is there a way to monitor actual disk transfers from command line without having to do manual timing?
Would running two CP command to copy 2 different set of files to two different targets suffice as a basic two thread test?
So long as you generate disk access through a file system and not hdparm.
Is there a way to monitor actual disk transfers from command line without having to do manual timing?
Like I said: iostat
iostat -m 1 iostat -x 1 iostat -d 1 iostat -xm 1
Choose whatever information/data you fancy.
In fact, let me just post some iostat output to settle this once and for all. I shall risk the annoyance of all by attaching the output of a 'iostat -m sda sdb md1 1' command.
This was generated by a single Linux client connected via cifs do a cp -r off the cifs remote share to local disk. The output was generated on the remote share box.
From the output, I guess one loses grounds for claiming that md raid1 only reads off one disk even if there are periods of time when it really does only read off one disk.
On Sat, Feb 21, 2009 at 11:42 PM, Chan Chung Hang Christopher < christopher.chan@bradbury.edu.hk> wrote:
Would running two CP command to copy 2 different set of files to two
different targets suffice as a basic two thread test?
So long as you generate disk access through a file system and not hdparm.
Is there a way to monitor actual disk transfers from command line without having to do manual timing?
Like I said: iostat
Thanks for the information. I checked iostat on one of my older servers
running off CentOS 5.0 (2.6.18-53.1.21.el5xen) which was also running md raid 1 and it also confirmed that the md raid 1 was getting reads from both member devices.
Although looking at it now, I think I really screwed up that installation, being my first, I had md running on top of LVM PV *slap forehead*
Kay Diederichs wrote:
Chan Chung Hang Christopher schrieb:
"md1 will read from both disk" is not true in general. RAID1 md reads from one disk only; it uses the other one in case the first one fails. No performance gain from multiple copies.
I beg to differ. I have disks in a raid1 md array and iostat -x 1 will show reads coming off both disks. Unless you do not have the multipath
look more carefully - with the current 2.6.18-9.1.22 kernel the bulk of the data are read from one of the disks
Hmm...right now I do not have a Centos 5 box handy. Come on you chums who have blasted me before about multipath. Prove him wrong with data please. I can only pull evidence off a Hardy box.
module loaded, md will read off both disks. Now whether md will read equally off both disks, that certainly will not be true in general.
You can easily see this for yourself by setting up a RAID1 from e.g. sda1 and sdb1 - /proc/mdstat is:
Personalities : [raid1] md1 : active raid1 sdb1[1] sda1[0] 104320 blocks [2/2] [UU]
and then comparing the output of hdparm -tT :
ROTFL.
How about using the proper tool (iostat) and generating some disk load instead?
hdparm -tT tests one type of disk access, other tools test other aspects. I gave the hdparm numbers because everyone can reproduce them. For RAID0 with two disks you do see - using e.g. hdparm - the doubling of performance from two disks. If you take the time to read (or do) RAID benchmarks you'll discover that Linux software RAID1 is about as fast as a single disk (and RAID0 with two disks is about twice the speed). It's as simple as that.
I beg to differ again since I did get combined throughput from a md raid1 device. I would have saved them iostat output to disk if I had known they would have some use. Anyway, I have got some numbers in my other post but on an Ubuntu box.
To get performance gain in RAID1 mode you need hardware RAID1.
Bollocks. The only area in which hardware raid has a significant performance advantage over software raid is raid5/6 given sufficient cache memory and processing power.
We were talking about RAID1; RAID5/6 is a different area. Linux software RAID1 is a safeguard against disk failure; it's not designed for speed increase. There is a number of things that could be improved in Linux software RAID; read performance of RAID1 is one of them - this _is_ why some hardware RAID1 adapters indeed are faster than software. Read http://kernelnewbies.org/KernelProjects/Raid1ReadBalancing - since the 2.6.25 kernel a simple alternating read is implemented, but that does not take the access pattern into account.
I have not read that yet but that is odd since I have been blasted by others before for doubting md raid1 doing multiple disk reads.
BTW, the Hardy box's kernel is 2.6.24-22-generic. I guess I need to try to generate some from an Intrepid box and see if I get better numbers.
So Linux software RAID1 is just mirroring - and it's good at that.
It has gotten good...no more having to sync from the beginning to end I believe...just like some hardware raid cards.
Chan Chung Hang Christopher wrote:
We were talking about RAID1; RAID5/6 is a different area. Linux software RAID1 is a safeguard against disk failure; it's not designed for speed increase. There is a number of things that could be improved in Linux software RAID; read performance of RAID1 is one of them - this _is_ why some hardware RAID1 adapters indeed are faster than software. Read http://kernelnewbies.org/KernelProjects/Raid1ReadBalancing - since the 2.6.25 kernel a simple alternating read is implemented, but that does not take the access pattern into account.
I have not read that yet but that is odd since I have been blasted by others before for doubting md raid1 doing multiple disk reads.
I know current centos alternates because I had a box with bad memory corrupt a raid1 filesystem and after fixing it and fsck'ing the disk, errors would slowly re-appear as they were hit on the alternate disk that previous fsck runs had not seen.
To really speed things up, you would want to try to avoid seeks. A simple alternating pattern may let the heads be in different places on small reads, but for a large file you will end up doing the same head motions on both drives (wasting the same time) as the reads alternate.
A good place to start comparing benchmark numbers for different RAID levels is http://linux-raid.osdl.org/index.php/Performance in particular the links given in section "Other benchmarks from 2007-2008"
HTH,
Kay
Kay Diederichs wrote:
A good place to start comparing benchmark numbers for different RAID levels is http://linux-raid.osdl.org/index.php/Performance in particular the links given in section "Other benchmarks from 2007-2008"
I like this bit of info from
http://linux-raid.osdl.org/index.php/Performance
Right under "Fresh benchmarking tools"
"To check out speed and performance of your RAID systems, do NOT use hdparm. It won't do real benchmarking of the arrays."
John R Pierce wrote:
Ray Van Dolson wrote:
Can't Linux LVM do mirroring? I swear I read that it could in the man page. Never have tried it however and you certainly can't set it up from disk druid in anaconda.
dunno. the word 'mirror occurs exactly once in the man page for lvm(8)
lvconvert -- Convert a logical volume from linear to mirror orsnapshot.
It's in the lvcreate manpage . . .
LVM mirroring seems rather quirky to me - I'd rather use md raid1 sets as phy vols.
On Feb 18, 2009, at 2:29 PM, Ray Van Dolson rayvd@bludgeon.org wrote:
On Wed, Feb 18, 2009 at 11:12:13AM -0800, John R Pierce wrote:
Scott Silva wrote:
You can make LVM over raid 1's in Disk Druid, but I don't think it will do raid 10. And you cannot boot from software raid 5 (yet).
a LVM over several raid 1's is effectively raid10 as LVM will stripe the volumes across the devices. It would be nice if LVM could do mirorring too (like LVM on AIX does) and was tighter integrated with the file system tools (again, like LVM on AIX... grow a LV and it grows the JFS thats sitting on it, transparently and online).
Can't Linux LVM do mirroring? I swear I read that it could in the man page. Never have tried it however and you certainly can't set it up from disk druid in anaconda.
Lvm can do mirroring, but of requires a third drive to do mirror logging which kind of defeats the whole raid1 concept, I guess if you have a VG of a plethora of drives this is no biggie.
And besides grub still after all these years does not support booting LVM volumes.
Jeez, you think with the Solaris folk able to boot off of ZFS now from grub we'd have the ability to boot off of LVM by now!
-Ross
On Wed, Feb 18, 2009 at 06:20:59PM -0500, Ross Walker wrote:
On Feb 18, 2009, at 2:29 PM, Ray Van Dolson rayvd@bludgeon.org wrote:
On Wed, Feb 18, 2009 at 11:12:13AM -0800, John R Pierce wrote:
Scott Silva wrote:
You can make LVM over raid 1's in Disk Druid, but I don't think it will do raid 10. And you cannot boot from software raid 5 (yet).
a LVM over several raid 1's is effectively raid10 as LVM will stripe the volumes across the devices. It would be nice if LVM could do mirorring too (like LVM on AIX does) and was tighter integrated with the file system tools (again, like LVM on AIX... grow a LV and it grows the JFS thats sitting on it, transparently and online).
Can't Linux LVM do mirroring? I swear I read that it could in the man page. Never have tried it however and you certainly can't set it up from disk druid in anaconda.
Lvm can do mirroring, but of requires a third drive to do mirror logging which kind of defeats the whole raid1 concept, I guess if you have a VG of a plethora of drives this is no biggie.
And besides grub still after all these years does not support booting LVM volumes.
Jeez, you think with the Solaris folk able to boot off of ZFS now from grub we'd have the ability to boot off of LVM by now!
Hehe, I guess the fact that we can achieve fairly similar results using md + LVM has made it a lower priority. :-/
Ray
a LVM over several raid 1's is effectively raid10 as LVM will stripe the volumes across the devices. It would be nice if LVM could do mirorring too (like LVM on AIX does) and was tighter integrated with the file system tools (again, like LVM on AIX... grow a LV and it grows the JFS thats sitting on it, transparently and online).
You heretic! Why are you asking for ZFS?
(btw the raid10 module is relatively new and is different from doing raid1+raid0 and is not supported by anaconda unlike raid1+0)
Subject: Re: [CentOS] 4 X 500 gb drives - best software raid config for abackup server?
Hi there, I am currently setting up a server that will house my
backups (simply
using rsync).
This system has 4 X 500 gb drives and I am looking to raid for max drive space and data safety. Performance is not so much a concern.
My experience with software raids in nil, so some of these may seem like pretty dumb questions.
I was thinking a raid 1 is probably sufficient.
Would it be best to raid 1 two drives each and LVM them together?
My next question would be about how to do this as I have
never done a
linux software raid.
You are in luck. I never did one before when I started too. I did a step by step procedure that covers adding and removing drives... For centos 5.x, with 4 drives. http://www.bobhoffman.com/wordpress/?page_id=44
This will take 2 drives into a raid 1, with LVM and all that. IT adds a third drive as a hot spare incase one fails. It adds a fourth as a standalone backup (good for a rync of the raid)
Just remember to wait for the drives to sync up before doing too much.
This is a step by step for newbs like me and is perfectly tailored for what you need. Let me know if you have any issues.
dnk wrote:
Hi there,
I am currently setting up a server that will house my backups (simply using rsync).
This system has 4 X 500 gb drives and I am looking to raid for max drive space and data safety. Performance is not so much a concern.
My experience with software raids in nil, so some of these may seem like pretty dumb questions.
I was thinking a raid 1 is probably sufficient.
Would it be best to raid 1 two drives each and LVM them together?
The configuration you want is a hybrid of RAID1 and RAID5. The RAID1 is because GRUB doesn't grok RAID5, but is OK with just your /boot partition in RAID1.
Make a RAID1 for your /boot partition as follows:
/dev/md0 - /dev/sda1, /dev/sdb1, /dev/sdc1 (S), /dev/sdd1 (S) (100 Mbytes)
Make a RAID5 as follows for a LVM partition using the rest of your available space as follows (just under 1500 Mbytes):
/dev/md1 - /dev/sda2, /dev/sdb2, /dev/sdc2, /dev/sdd2
Create a LVM partition on /dev/md1 and carve out your / and swap partitions from it.
My next question would be about how to do this as I have never done a linux software raid.
-
Bob Hoffman -
Chan Chung Hang Christopher -
Christopher Chan
-
dnk -
Ian Forde -
Jerry Franz -
John Doe -
John Plemons -
John R Pierce -
Kay Diederichs -
Les Mikesell -
Noob Centos Admin -
Ray Van Dolson -
Rob Kampen -
Ross Walker -
Scott Silva -
Toby Bluhm