Hi everyone,
I'm looking into setting up a SATA hardware raid, probably 5 to use with CentOS 4. I chose hardware raid over software mostly because I like the fact that the raid is transparent to the OS.
Does anyone know of any SATA controllers that are well tested for this sort of usage?
From what I can tell from googling, this is more or less where RHEL stands: Red Hat Enterprise Linux version 3 and the current version of Fedora support the following SATA chipsets:
Intel's ICH5 SATA chipset Silicon Image's SATA chipset
Does CentOS4 add anything to this as it is based on 2.6 kernel?
My question upon reading what I found on Google, is if it is true hardware raid, shouldn't the OS not be able to tell it's raid at all? I'm assuming that the chipsets listed above are driver based hardware raid? what I am after is a raid array based on true hardware raid such that the OS see's just one drive, and the hardware firmware handles any mirror/striping.
Any suggestions?
regards
Franki
On Sat, 16 Apr 2005 at 1:11am, Franki wrote
I'm looking into setting up a SATA hardware raid, probably 5 to use with CentOS 4. I chose hardware raid over software mostly because I like the fact that the raid is transparent to the OS.
Does anyone know of any SATA controllers that are well tested for this sort of usage?
IMO, 3ware is far and away the best choice for true hardware SATA RAID. The 3w-xxxx driver (for the 8000 series boards) has been in the kernel for a *long* time, and 3w-9xxx driver (for the 9000 series) has been in a while as well. There are some better performers out there at the moment (Areka, for example), but their drivers are unproven.
FWIW, I currently have 4TB (2 servers) worth of 3ware based storage in production, and just took delivery of a new 5.5TB server.
Joshua Baker-LePain wrote:
On Sat, 16 Apr 2005 at 1:11am, Franki wrote
Does anyone know of any SATA controllers that are well tested for this sort of usage?
IMO, 3ware is far and away the best choice for true hardware SATA RAID. The 3w-xxxx driver (for the 8000 series boards) has been in the kernel for a *long* time, and 3w-9xxx driver (for the 9000 series) has been in a while as well. There are some better performers out there at the moment (Areka, for example), but their drivers are unproven.
FWIW, I currently have 4TB (2 servers) worth of 3ware based storage in production, and just took delivery of a new 5.5TB server.
I would also like to put my 2c in here.
I've tried every SATA controller out there, and 3ware is the only one you should be considering for any serious linux application. I've had cards in production for 4 years, and never had a single problem or failure.
They just work. What more do you want?
Jonathan
Am Fr, den 15.04.2005 schrieb Franki um 19:11:
I'm looking into setting up a SATA hardware raid, probably 5 to use with CentOS 4. I chose hardware raid over software mostly because I like the fact that the raid is transparent to the OS.
Does anyone know of any SATA controllers that are well tested for this sort of usage?
From what I can tell from googling, this is more or less where RHEL stands: Red Hat Enterprise Linux version 3 and the current version of Fedora support the following SATA chipsets:
Intel's ICH5 SATA chipset Silicon Image's SATA chipset
Those are no hardware RAID controllers. They are 'winraid' / 'fake raid' as it is BIOS supported software RAID. If you want real hardware RAID have a look at 3Ware's products.
Does CentOS4 add anything to this as it is based on 2.6 kernel?
Kernel 2.6 too supports 3Ware controllers well.
My question upon reading what I found on Google, is if it is true hardware raid, shouldn't the OS not be able to tell it's raid at all?
Right, the system sees 1 drive rather than the independent drives. Thats the difference between hardware RAID and software RAID.
I'm assuming that the chipsets listed above are driver based hardware raid? what I am after is a raid array based on true hardware raid such that the OS see's just one drive, and the hardware firmware handles any mirror/striping.
Correct :)
Any suggestions?
Yes, 3Ware - from small controllers to those with RAID5 capability and several SATA connectors. ICP Vortex may be a good choice too. The stories about Adaptec controllers do not invite to be interested in them.
Franki
Alexander
On Sat, 2005-04-16 at 01:11 +0800, Franki wrote:
Hi everyone,
I'm looking into setting up a SATA hardware raid, probably 5 to use with CentOS 4. I chose hardware raid over software mostly because I like the fact that the raid is transparent to the OS.
Does anyone know of any SATA controllers that are well tested for this sort of usage?
From what I can tell from googling, this is more or less where RHEL stands: Red Hat Enterprise Linux version 3 and the current version of Fedora support the following SATA chipsets:
Intel's ICH5 SATA chipset Silicon Image's SATA chipsetDoes CentOS4 add anything to this as it is based on 2.6 kernel?
My question upon reading what I found on Google, is if it is true hardware raid, shouldn't the OS not be able to tell it's raid at all? I'm assuming that the chipsets listed above are driver based hardware raid? what I am after is a raid array based on true hardware raid such that the OS see's just one drive, and the hardware firmware handles any mirror/striping.
I have used the 6 port LSI Megaraid SATA controller for an application using an RHEL rebuild (Rocks) with a 2.4 kernel. It simply uses the same modules as the SCSI flavor (e.g some of the Dell PERC 4 line). It has worked like a champ (so far), and it wasn't too outrageous priced.
----- Original Message ----- From: "Franki" franki@htmlfixit.com Sent: Friday, April 15, 2005 10:11 AM Subject: [CentOS] Serial ATA hardware raid.
I'm looking into setting up a SATA hardware raid, probably 5 to use with CentOS 4. I chose hardware raid over software mostly because I like the fact that the raid is transparent to the OS.
Does anyone know of any SATA controllers that are well tested for this sort of usage?
[...]
Any suggestions?
From experience, 3Ware has the best support for hardware SATA RAID on Linux,
whether on kernel 2.4 or 2.6. Their drivers have been included in most distros for a very long time, and even the latest 9xxx cards are fully supported out of the box, or with drivers you can freely download from 3Ware web site. We use their cards on several production servers, including a few that have 12 SATA drives for massive 3TB storage solutions. I don't recall ever having problems with their cards or support.
Chris
centos.org_list@000a.com wrote:
From experience, 3Ware has the best support for hardware SATA RAID on Linux,
whether on kernel 2.4 or 2.6. Their drivers have been included in most distros for a very long time, and even the latest 9xxx cards are fully supported out of the box, or with drivers you can freely download from 3Ware web site. We use their cards on several production servers, including a few that have 12 SATA drives for massive 3TB storage solutions. I don't recall ever having problems with their cards or support.
Chris
What he said. 8-) 3Ware is the way to go. I've been using thier cards w/linux for 5+ years w/nary a problem.
Cheers,
C
centos-bounces@centos.org wrote on 15.04.2005 19:11:09:
I'm looking into setting up a SATA hardware raid, probably 5 to use with
CentOS 4. I chose hardware raid over software mostly because I like the fact that the raid is transparent to the OS.
Does anyone know of any SATA controllers that are well tested for this sort of usage?
3ware (7500, 8500, 9500) is IMHO best bet on all distros including CentOS 4. I also run it on an LSI SATA controller, but I cannot remember the exact model. But I can find it if you want to know. :)
Regards, Harald
centos-bounces@centos.org wrote on 15.04.2005 19:11:09:
Does anyone know of any SATA controllers that are well tested for this sort of usage?
Just have to mention it while I remember. If you decide on the 3ware 9500s controller, be aware that you HAVE to do some manual tuning. It performs like crap out of the box.
Regards, Harald
Harald Finnås wrote:
centos-bounces@centos.org wrote on 15.04.2005 19:11:09:
Does anyone know of any SATA controllers that are well tested for this sort of usage?
Just have to mention it while I remember. If you decide on the 3ware 9500s controller, be aware that you HAVE to do some manual tuning. It performs like crap out of the box.
Regards, Harald
Thanks for all the tips guys, I'm now wandering around 3wares site. http://www.3ware.com/products/serial_ata8000.asp
What I get is determined by what is available in Western Australia, though I suppose these are small enough that I can get them shipped pretty cheap however if one dies I'd like to be able to get it swapped out quickly so I guess I should stick with one of my local wholesalers.
Looking at this page: http://www.auspcmarket.com.au/index.php?redir=http://www.auspcmarket.com.au/...
The 3ware 9500 goes for $1023 AUD. once I add some fast drives to that this thing is starting to look like an expensive upgrade. I've just checked my wholesaler pricelists and I don't see any 3ware kit. (bummer) so if I buy one, I'll be getting retail and still having to ship it 5000km (from Sydney to Perth.)
My main wholesaler sells Adaptec stuff, and I can get this one: http://www.techbuy.com.au/products/34526/I_OCARDS_HARDDRIVECONTROLLERS/Adapt...
For $450 it is a 4 port controller and has onboard 64MB ECC cache.
But Alexander on the list here said that adaptec is questionable for Raid cards.. The last time I had an Adaptec anything was a 2940UW Scsi controller and that thing gave me years of good service.
Not sure what to chase up now. :-( Does anyone have any experiance with the above adaptec card?
rgds
Franki
On Sat, 16 Apr 2005 at 3:40am, Franki wrote
Looking at this page: http://www.auspcmarket.com.au/index.php?redir=http://www.auspcmarket.com.au/...
The 3ware 9500 goes for $1023 AUD. once I add some fast drives to that this thing is starting to look like an expensive upgrade.
That's an 8 port card. The 4 port model goes for $671 on that site, which is more comparable to the 4-port adaptec card you mentioned.
Hi,
On Fri, Apr 15, 2005 at 09:13:17PM +0200, Harald Finnås wrote:
centos-bounces@centos.org wrote on 15.04.2005 19:11:09:
Does anyone know of any SATA controllers that are well tested for this sort of usage?
Just have to mention it while I remember. If you decide on the 3ware 9500s controller, be aware that you HAVE to do some manual tuning. It performs like crap out of the box.
This is as good insertion point as anything else, for what i am going to say.
3ware is working like a champ, but slowly. The tuning won't make it magically go over 100MB/s sustained writes. Random I/O sucks (what i've seen) for any SATA-setup. Even for /dev/mdX. Puny oldish A1000 can beat those with almoust factor of ten for random I/O, but being limited to max. 40MB/s transfers by it's interface (UW/HVD).
But what i am going to say is that for my centos devel work (as in my NFS-server), i just recently moved my 1.6TB raid under /dev/md5 with HighPoint RocketRaid1820. I don't care that NOT being hardware RAID. The /dev/mdX beats the 3ware 9500S-8 formerly used hands down when you do have 'spare CPU cycles to let kernel handle the parity operations'.
http://www.supermicro.com/products/accessories/addon/DAC-SATA-MV8.cfm
should be even cheaper 8-port solution, but those RocketRAIDs are available on like in every store.
Google reveals that there is souce driver for these Marvel-chips (mvsata340.zip) like in
http://www2.abit.com.tw/page/en/server/server_detail.php?pMODEL_NAME=SU-2S&a...
That driver isn't exactly GPL, but it has been working quite good past month or two while i've been on my 'testing pahse' with this.
With that Supermicro board, one doesn't even have annoying BIOS problems at POST (that damn 128k BIOS init window + order of card detection etc. problems which won't always result the system you'd like to have :).
So i am not saying the 3ware isn't good solution, i am just saying that at least for me there are better solutions which gets me cheaper more compatible and faster solution. The metadevice is just something one can toss in to any hardware as it's in kernel. The 3ware needs 3ware for replacement if RAID5(0) is used. I've told that RAID1 on 3ware is just hardware mirror and those disks can be used on any controller, but i have not personally verified it.
Then few words about this '3ware driver'. How i see it, it's just a few lines wrapping linux SCSI-layer to 3ware firmware and not even doing it too good. There has been constantly talk about that the 3ware firmware would be the reason of bad performance for example CentOS-3/ext3 + 3ware. Noone out of 3ware can do anything about it and it's as it is. With kernel solution, every line of code is there and anyone can study it and find the problem if it should exist.
The now used metaformat for devices is just something which is plug'n'play. I just recently had 'a pack of disk laying around in random order' which had been 8 disk /dev/md5 and RAID5. Plugged those in a machine and found out that it was re-sunching the raid before i logged in and i just tossed those disks in w/o any knowledge about the previous order as i wasn't even going to preserve it. So the linux software implementation has grown to be pretty damn good and neat.
But then again. I need to get another cup of coffee to really wake up :P
my .02 euros
Harald Finnås wrote:
Just have to mention it while I remember. If you decide on the 3ware 9500s controller, be aware that you HAVE to do some manual tuning. It performs like crap out of the box.
I was wondering if you could expand on that. What type of array(s) are you using, which controller and any specific settings?
Avtar Gill wrote on 16.04.2005 08:54:50:
Just have to mention it while I remember. If you decide on the 3ware 9500s controller, be aware that you HAVE to do some manual tuning. It performs like crap out of the box.
I was wondering if you could expand on that. What type of array(s) are you using, which controller and any specific settings?
If memory serves, I had the worst performance on aa 7506-8 running R5 and MDK 10.1. Write performance was extremenly slow, so the box would use all memory for cache almost immidiately. This resulted in very high CPU, and the machine became unresponsive. After applying:
sysctl -w "vm.bdflush=0 500 0 0 500 3000 0 20 0"
it was back to normal. This works with 2.4 kernels only. With 2.6 this improves performance:
blockdev --setra 16384 /dev/sda
I'm pretty sure I found both these setting in the 3ware knowledgebase.
Regards, Harald
-
Alexander Dalloz -
Avtar Gill -
centos.org_list＠000a.com -
Chris Mauritz -
Franki -
Harald Finnås -
Jonathan -
Joshua Baker-LePain -
Pasi Pirhonen -
Sean O'Connell