Hello listmates,
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
Thanks.
Boris.
On 4/13/2010 1:05 PM, Boris Epstein wrote:
Hello listmates,
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
Thanks.
Boris. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Chassis - CSE-836A-R1200B Supermicro SC836 A-R1200B - Rack-mountable - 3U - SATA/SAS - hot-swap - power supply 1200 Watt
RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL RAID 0/1/5/6/10/50 16CH SATA II PCIE 256MB ECC DDR2 - PCI Express x8 - Up to 300MBps - 4 x SATA x4 Serial ATA/300 - Serial ATA
BBU Module for RAID card - 3ware BBU-MODULE-03
Pick the cpu(s) and motherboard to fit the chassis. Obviously go with ECC ram and ONLY enterprise grade hard drives. To ensure compatibility check with 3ware to see which drives they recommend. Areca RAID cards will get you a little better performance but the module for the 9650SE series of 3ware cards is included with the Centos kernel. Getting the Areca driver going is a bit more work, but nothing that would be considered a huge hurdle for a competent sysadmin. Also, if you're looking for advice on Areca products call their Tekram contact in the USA. Their other distributors have been less than stellar on answering pre-sales questions.
On Tue, Apr 13, 2010 at 1:14 PM, Ryan Manikowski ryan@devision.us wrote:
On 4/13/2010 1:05 PM, Boris Epstein wrote:
Hello listmates,
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
Thanks.
Boris. _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Chassis - CSE-836A-R1200B Supermicro SC836 A-R1200B - Rack-mountable - 3U - SATA/SAS - hot-swap - power supply 1200 Watt
RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL RAID 0/1/5/6/10/50 16CH SATA II PCIE 256MB ECC DDR2 - PCI Express x8 - Up to 300MBps - 4 x SATA x4 Serial ATA/300 - Serial ATA
BBU Module for RAID card - 3ware BBU-MODULE-03
Pick the cpu(s) and motherboard to fit the chassis. Obviously go with ECC ram and ONLY enterprise grade hard drives. To ensure compatibility check with 3ware to see which drives they recommend. Areca RAID cards will get you a little better performance but the module for the 9650SE series of 3ware cards is included with the Centos kernel. Getting the Areca driver going is a bit more work, but nothing that would be considered a huge hurdle for a competent sysadmin. Also, if you're looking for advice on Areca products call their Tekram contact in the USA. Their other distributors have been less than stellar on answering pre-sales questions.
-- Ryan Manikowski
]] Devision Media Services LLC [[ www.devision.us ryan@devision.us | 716.771.2282
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Thanks Ryan!
So, basically, I take it you've had positive experience with that Supermicro box. It does look good and seems fairly inexpensive, too.
Boris.
On Tue, Apr 13, 2010 at 1:14 PM, Ryan Manikowski ryan@devision.us wrote:
On 4/13/2010 1:05 PM, Boris Epstein wrote:
Hello listmates,
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
Thanks.
Boris. _______________________________________________ CentOS mailing listCentOS@centos.orghttp://lists.centos.org/mailman/listinfo/centos
Chassis - CSE-836A-R1200B Supermicro SC836 A-R1200B - Rack-mountable - 3U - SATA/SAS - hot-swap - power supply 1200 Watt
RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL RAID 0/1/5/6/10/50 16CH SATA II PCIE 256MB ECC DDR2 - PCI Express x8 - Up to 300MBps - 4 x SATA x4 Serial ATA/300 - Serial ATA
BBU Module for RAID card - 3ware BBU-MODULE-03
Pick the cpu(s) and motherboard to fit the chassis. Obviously go with ECC ram and ONLY enterprise grade hard drives. To ensure compatibility check with 3ware to see which drives they recommend. Areca RAID cards will get you a little better performance but the module for the 9650SE series of 3ware cards is included with the Centos kernel. Getting the Areca driver going is a bit more work, but nothing that would be considered a huge hurdle for a competent sysadmin. Also, if you're looking for advice on Areca products call their Tekram contact in the USA. Their other distributors have been less than stellar on answering pre-sales questions.
-- Ryan Manikowski
]] Devision Media Services LLC [[ www.devision.us ryan@devision.us | 716.771.2282
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Ryan's hardware recommendations are good. But I wouldn't run a RAID5 volume that large, software or hardware. It's just too risky as rebuilds will take days and the chances of hitting a non recoverable read error would be near 100% on a volume that size.
Either run multiple smaller RAID5's and use LVM to manage the volumes which the OS will use or choose a better RAID layout. RAID6 or RAID10 are much better choices these days. -- David
On 4/13/2010 1:19 PM, David Miller wrote:
<snip>
Ryan's hardware recommendations are good. But I wouldn't run a RAID5 volume that large, software or hardware. It's just too risky as rebuilds will take days and the chances of hitting a non recoverable read error would be near 100% on a volume that size.
Either run multiple smaller RAID5's and use LVM to manage the volumes which the OS will use or choose a better RAID layout. RAID6 or RAID10 are much better choices these days. -- David
With the config mentioned above it would give the flexibility to run RAID10 with a resulting data store of just under 14TB (8x RAID1 stripe using 2TB drives).
Choice of RAID implementation (specifically RAID5) could be an impediment to performance as noted above. Always good to have input from more than one source. At that level of storage looking into spending a bit more for redundancy (drbd/pacemkaker/heartbeat) may be a worthwhile investment as well.
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
This comes up often:) Things to keep in mind when using large discs if you suffer a failure, while rebuilding in a degraded state it's not impossible or unlikely to drop another disc loosing it all.
Rebuild times especially on busy arrays with large discs take lots of time...
On Tue, Apr 13, 2010 at 2:30 PM, Joseph L. Casale jcasale@activenetwerx.com wrote:
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
This comes up often:) Things to keep in mind when using large discs if you suffer a failure, while rebuilding in a degraded state it's not impossible or unlikely to drop another disc loosing it all.
Rebuild times especially on busy arrays with large discs take lots of time... _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Thanks Joseph!
In your opinion, if we are talking about 15 2-TB disks what does "lots of time" translate to?
Boris.
Boris Epstein wrote:
On Tue, Apr 13, 2010 at 2:30 PM, Joseph L. Casale jcasale@activenetwerx.com wrote:
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
This comes up often:) Things to keep in mind when using large discs if you suffer a failure, while rebuilding in a degraded state it's not impossible or unlikely to drop another disc loosing it all.
Rebuild times especially on busy arrays with large discs take lots of time... _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Thanks Joseph!
In your opinion, if we are talking about 15 2-TB disks what does "lots of time" translate to?
well, IF your controller totally screams and can rebuild the drives at wire speeds with full overlap, you'll be reading 7 * 2TB of data at around 100MB/sec average and writing the XOR of that to the 8th drive in a 8 spindle raid5 (14tb total). just reading one drive at wirespeed is 2000,000MB / 100MB == 20,000 seconds, or about 5.5 hours, so thats about the shortest it possibly could be done.
if you have any access activity during this rebuild, you can figure on nearly doubling that before you even get warm. if the controller can't actually move 800MB/sec internally (believe me, that ain't easy), then you can double or quadruple it a few more times. This 5.5 hours could easily stretch to a week.
ANY previously unnoticed bad sector on those other 7 drives renders the whole pile bad, so doing weekly raid sweeps is probably a good thing, except THOSE will take some multiple of 5.5 hours when run in the background.
John R Pierce wrote:
well, IF your controller totally screams and can rebuild the drives at wire speeds with full overlap, you'll be reading 7 * 2TB of data at around 100MB/sec average and writing the XOR of that to the 8th drive in a 8 spindle raid5 (14tb total). just reading one drive at wirespeed is 2000,000MB / 100MB == 20,000 seconds, or about 5.5 hours, so thats about the shortest it possibly could be done.
More likely your looking at 24+ hours, because really no disk system is going to read your SATA disks at 100MB/second. If your really lucky perhaps you can get 10MB/second.
With the fastest RAID controllers in the industry my own storage array(which does heavy amounts of random I/O) averages about 2.2MB/second for a SATA disk, with peaks at around 4MB/second.
Our previous storage array averaged about 4-6 hours to rebuild a RAID 5 12+1 array with 146GB 10k RPM disks, on an array that was in excess of 90% idle. Rebuilding a 400GB SATA-I array often took upwards of 48 hours.
nate
On Apr 13, 2010, at 11:57 AM, nate wrote:
John R Pierce wrote:
well, IF your controller totally screams and can rebuild the drives at wire speeds with full overlap, you'll be reading 7 * 2TB of data at around 100MB/sec average and writing the XOR of that to the 8th drive in a 8 spindle raid5 (14tb total). just reading one drive at wirespeed is 2000,000MB / 100MB == 20,000 seconds, or about 5.5 hours, so thats about the shortest it possibly could be done.
More likely your looking at 24+ hours, because really no disk system is going to read your SATA disks at 100MB/second. If your really lucky perhaps you can get 10MB/second.
With the fastest RAID controllers in the industry my own storage array(which does heavy amounts of random I/O) averages about 2.2MB/second for a SATA disk, with peaks at around 4MB/second.
Our previous storage array averaged about 4-6 hours to rebuild a RAID 5 12+1 array with 146GB 10k RPM disks, on an array that was in excess of 90% idle. Rebuilding a 400GB SATA-I array often took upwards of 48 hours.
nate
For a "real life" example, we have a 3 year old 12x 1TB SATA box using an Adaptec RAID controller, doing RAID 6 that takes about 3 days to rebuild the array each time a drive fails. Which, to date, has happened 10 times... (Fortunately, this is only a BackupPC box.)
FWIW, we've not experienced a second drive failure during the rebuild process, yet. But we have had drives fail within a few weeks of each other, so it's probably going to happen one of these days..
-- Don Krause Head Systems Geek, Waver of Deceased Chickens. Optivus Proton Therapy, Inc. www.optivus.com "This message represents the official view of the voices in my head."
Those drives are likely fading out of the array because they aren't meant to be in arrays in the first place, Adaptec has told us that if you use consumer drives with their cards you are operating at your own risk.
-Drew
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Don Krause Sent: Tuesday, April 13, 2010 3:20 PM To: CentOS mailing list Subject: Re: [CentOS] 12-15 TB RAID storage recommendations
On Apr 13, 2010, at 11:57 AM, nate wrote:
John R Pierce wrote:
well, IF your controller totally screams and can rebuild the drives at wire speeds with full overlap, you'll be reading 7 * 2TB of data at around 100MB/sec average and writing the XOR of that to the 8th drive in a 8 spindle raid5 (14tb total). just reading one drive at wirespeed is 2000,000MB / 100MB == 20,000 seconds, or about 5.5 hours, so thats about the shortest it possibly could be done.
More likely your looking at 24+ hours, because really no disk system is going to read your SATA disks at 100MB/second. If your really lucky perhaps you can get 10MB/second.
With the fastest RAID controllers in the industry my own storage array(which does heavy amounts of random I/O) averages about 2.2MB/second for a SATA disk, with peaks at around 4MB/second.
Our previous storage array averaged about 4-6 hours to rebuild a RAID 5 12+1 array with 146GB 10k RPM disks, on an array that was in excess of 90% idle. Rebuilding a 400GB SATA-I array often took upwards of 48 hours.
nate
For a "real life" example, we have a 3 year old 12x 1TB SATA box using an Adaptec RAID controller, doing RAID 6 that takes about 3 days to rebuild the array each time a drive fails. Which, to date, has happened 10 times... (Fortunately, this is only a BackupPC box.)
FWIW, we've not experienced a second drive failure during the rebuild process, yet. But we have had drives fail within a few weeks of each other, so it's probably going to happen one of these days..
-- Don Krause Head Systems Geek, Waver of Deceased Chickens. Optivus Proton Therapy, Inc. www.optivus.com "This message represents the official view of the voices in my head."
_______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
They weren't supposed to be consumer drives. The box was provided by the vendor of a disk-disk-tape backup system.
They are Western Digital Enterprise RE2-GP drives.
I wouldn't purchase them again, thanks for asking...
Not that this provides any real info to the OP, other than the time it takes to rebuild the array.
=Don=
On Apr 13, 2010, at 12:24 PM, Drew Weaver wrote:
Those drives are likely fading out of the array because they aren't meant to be in arrays in the first place, Adaptec has told us that if you use consumer drives with their cards you are operating at your own risk.
-Drew
-----Original Message----- From: centos-bounces@centos.org [mailto:centos-bounces@centos.org] On Behalf Of Don Krause Sent: Tuesday, April 13, 2010 3:20 PM To: CentOS mailing list Subject: Re: [CentOS] 12-15 TB RAID storage recommendations
On Apr 13, 2010, at 11:57 AM, nate wrote:
John R Pierce wrote:
well, IF your controller totally screams and can rebuild the drives at wire speeds with full overlap, you'll be reading 7 * 2TB of data at around 100MB/sec average and writing the XOR of that to the 8th drive in a 8 spindle raid5 (14tb total). just reading one drive at wirespeed is 2000,000MB / 100MB == 20,000 seconds, or about 5.5 hours, so thats about the shortest it possibly could be done.
More likely your looking at 24+ hours, because really no disk system is going to read your SATA disks at 100MB/second. If your really lucky perhaps you can get 10MB/second.
With the fastest RAID controllers in the industry my own storage array(which does heavy amounts of random I/O) averages about 2.2MB/second for a SATA disk, with peaks at around 4MB/second.
Our previous storage array averaged about 4-6 hours to rebuild a RAID 5 12+1 array with 146GB 10k RPM disks, on an array that was in excess of 90% idle. Rebuilding a 400GB SATA-I array often took upwards of 48 hours.
nate
For a "real life" example, we have a 3 year old 12x 1TB SATA box using an Adaptec RAID controller, doing RAID 6 that takes about 3 days to rebuild the array each time a drive fails. Which, to date, has happened 10 times... (Fortunately, this is only a BackupPC box.)
FWIW, we've not experienced a second drive failure during the rebuild process, yet. But we have had drives fail within a few weeks of each other, so it's probably going to happen one of these days..
-- Don Krause Head Systems Geek, Waver of Deceased Chickens. Optivus Proton Therapy, Inc. www.optivus.com "This message represents the official view of the voices in my head."
They weren't supposed to be consumer drives. The box was provided by the vendor of a disk-disk-tape backup system.
They are Western Digital Enterprise RE2-GP drives.
I wouldn't purchase them again, thanks for asking...
<snip> Dunno 'bout them, but I *really* don't like Seagate Barracudas. Three times now, in the last 15 years, they've shoved new models out that did *not* meet q/c, and it took them a year or two to fix the problems.
mark
On Tuesday 13 April 2010, Drew Weaver drew.weaver@thenap.com wrote:
Those drives are likely fading out of the array because they aren't meant to be in arrays in the first place, Adaptec has told us that if you use consumer drives with their cards you are operating at your own risk.
Every hard drive dies. It's just a matter of when.
Joseph L. Casale wrote:
Rebuild times especially on busy arrays with large discs take lots of time...
Unless you have a good storage system..
a blog entry I wrote last year: http://www.techopsguys.com/2009/11/24/81000-raid-arrays/
Another one where I ripped into Equallogic's claims: http://www.techopsguys.com/2010/03/26/enterprise-equallogic/
Just checked my array again, nearly 200,000 RAID arrays on it, makes for a massively parallel many:many RAID rebuild for fast recovery times with *no* service impact.
Course this stuff may be out of the OP's budget, but just keep in mind that there are such systems on the market.
IBM XIV is another such system, though it's scalability is too limited to be useful IMO (~180 drives max).
You'd have to pull my toenails out with a rusty pair of pliers before I go back to crap storage.
nate
Unless you have a good storage system..
a blog entry I wrote last year: http://www.techopsguys.com/2009/11/24/81000-raid-arrays/
Another one where I ripped into Equallogic's claims: http://www.techopsguys.com/2010/03/26/enterprise-equallogic/
Lol, Nate... The op was looking at spending a few grand, not a few million you show off:)
Joseph L. Casale wrote:
Unless you have a good storage system..
a blog entry I wrote last year: http://www.techopsguys.com/2009/11/24/81000-raid-arrays/
Another one where I ripped into Equallogic's claims: http://www.techopsguys.com/2010/03/26/enterprise-equallogic/
Lol, Nate... The op was looking at spending a few grand, not a few million you show off:)
million? Nowhere close to that, you can get a 12-15TB system(raw) in the ~$130-150k range (15k RPM). If you want SATA instead say $80k.
Few million and you can get a world record breaker array with more than a thousand drives(15k RPM) and loaded with all the software they have.
The capabilities of the system is the same from the low end to the high end the only difference is scale really.
nate
2010/4/13 nate centos@linuxpowered.net:
Joseph L. Casale wrote:
Unless you have a good storage system..
a blog entry I wrote last year: http://www.techopsguys.com/2009/11/24/81000-raid-arrays/
Another one where I ripped into Equallogic's claims: http://www.techopsguys.com/2010/03/26/enterprise-equallogic/
Lol, Nate... The op was looking at spending a few grand, not a few million you show off:)
million? Nowhere close to that, you can get a 12-15TB system(raw) in the ~$130-150k range (15k RPM). If you want SATA instead say $80k.
err. you can get hitachi sms 100 with sata drives for about 9000e including 3 year maintenance.
-- Eero
Eero Volotinen wrote:
err. you can get hitachi sms 100 with sata drives for about 9000e including 3 year maintenance.
Yes, and you get what you pay for with that..
As I mentioned earlier myself I won't go back to crap storage after seeing the light..
Even Hitachi AMS 2k series doesn't compare.
nate
2010/4/13 Boris Epstein borepstein@gmail.com:
Hello listmates,
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
how about hitachi SMS 100 ? it is about that size and cost effective iscsi solution?
-- Eero
On Tue, 13 Apr 2010, Boris Epstein wrote:
Hello listmates,
I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc?
Thanks.
Boris.
Smaller volumes is best, but really it depends on your I/O type as well. I have 15TB volumes loaded with medical imaging data that happily run and fsck just fine. We've had a couple of disk failures and the MD 3000 and MD1000 units handled this just fine taking around 26 hours to sync 1TB drives. The file system here is XFS
On the other hand, I have natural language data sets which are millions of small files residing on a 4.5TB EXT4 file system. This file system has had a problem and to this day I still cannot perform a file system check to correct the errors because the e4fsck program chews up more than 42GB of memory and then dies. For details check out.
https://bugzilla.redhat.com/show_bug.cgi?id=570639
What I'm trying to say is, understand your usage patterns. Large streaming files is far less intensive on the controller then millions of small files.
Understand your hardware and what it is capable of in each configuration. RAID-0 vs 5 vs 6 vs 10. It is incredibly important.
Understand your file system. Figure out what file system works best for your workload, how it functions and how the underlying hardware needs to be configured to maximize throughput.
That is all for now. ;)
-
Alan Hodgson -
Boris Epstein -
David Miller -
Don Krause -
Drew Weaver -
Eero Volotinen -
James A. Peltier -
John R Pierce -
Joseph L. Casale -
m.roth＠5-cent.us -
nate -
Ryan Manikowski