Chris Mauritz wrote:

Lee W wrote:

I don't have any experience with the Promise card, but I can highly recommend any recent vintage 3Ware RAID card. They're not terribly expensive and the new 9500 4 port cards should be tiny enough to easily fit in a mini-ITX case. That card plus 4 400gig drives would give you a 4 disk RAID 5 array (with very good read/write performance) and well over a terabyte of usable storage. It wasn't long ago that a 10gig drive was as big as a shoebox and now you can fit over a terabyte of redundant storage and a fairly powerful computer in the same form factor. We live in interesting times. Cheers,

C

---

CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos

Hi Chris,

Thanks for the very prompt reply. Any ideas where I can get them?

My other searches mentioned a few 3Ware cards but they don't seem to be available anywhere in the UK.

Regards

Lee

On Tue, 10 May 2005 at 8:36pm, Lee W wrote

Okay I'm getting a bit over my head now. Anyone see any problems putting 2 of these in the same machine. After a few other comments I was thinking of trying 2 x RAID-1's.

Most (all but one, actually) of my 3ware based servers have multiple cards in 'em. My most recent has two 9500s-12 boards in it (on separate PCI busses), each board set to RAID 5 w/ a hot spare, and I do a software RAID0 across the two arrays. Performance (tiobench) graphs here:

http://www.duke.edu/~jlb17/tioread.png http://www.duke.edu/~jlb17/tiowrite.png

In short, it goes fast.

Hi

A word of caution regarding the 3ware cards -- this bug hasn't yet been solved:

- Extremely high iowait with 3Ware array and moderate disk activity https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434

Chris

Hi,

On Wed, May 11, 2005 at 12:23:30PM +0100, John Moylan wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

That's for the PATA cards. I don't think that the 9500-S cards are effected by this.

BTW, I have noticed that RHEL 3U5 Beta has a lot of additions and improvements to the 3ware drivers in their changlist.

J

Chris Croome wrote: | Hi | | | - Extremely high iowait with 3Ware array and moderate disk activity | https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434 |

Please don't TOP-post.

But the fact is that even 9500S-x is affected with RHEL3-level kernel. SMP will trigger easily the 'sluggish behaviour' with ext3. I've not veen able to trigger this with UP kernel tho. Patching XFS in and using that, won't trigger any anomalities anymore.

The problem isn't actually high I/O wait. It's perfectly OK to have high I/O wait. The problem is that the write performance drops below 1MB/s and stays there as long as the constant writing continues. Sometimes it can go allright _some minutes_, but it'll manifest itself really easy.

On Wed, 2005-05-11 at 11:48 +0100, Chris Croome wrote:

Hi A word of caution regarding the 3ware cards -- this bug hasn't yet been solved:

• Extremely high iowait with 3Ware array and moderate disk activity https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=121434

Chris

It's not a "bug," it's the design of the 3Ware 8506 cards. Kernels change defaults regularly, which means people need to be explicitly setting the most optimal kernel parameters in their /etc/rc.d/rc.local or similar once they find them.

3Ware's site has all sorts of recommendations.

The 3Ware cards will queue up more I/O operations and off-load so much from the kernel -- far more than any other card I've seen. The problem is when implementing RAID-5 on a 7000/8000 series, the measly 2-4MB of SRAM (static RAM) overflows on even a moderate number of writes. The I/O then stalls. The key is to tweak the I/O parameters of the kernel to handle this limitation of the 3Ware 7000/8000 series.

The 3Ware 7000/8000 series cards are "storage switches" and _not_ "buffering controllers."

It's like if your NFS/Samba server could only store files in its CPU cache and didn't have main memory, before sending information to disk. SMB operations would quickly stall and have to wait on the disk writes. The majority of other controllers have the opposite problem. They are like having a CPU with no cache, just a lot of memory. I know this is an over-simplification, but that's the best analogy I can come up with.

The 3Ware 9000 series adds a good amount of DRAM for more buffering operations, such as RAID-5 writes. But they are new, and the drivers are still maturing.

On Wed, 11 May 2005 at 3:06pm, Chris Croome wrote

On Wed 11-May-2005 at 08:40:33AM -0500, Bryan J. Smith wrote:

...

The 3Ware 9000 series adds a good amount of DRAM for more buffering operations, such as RAID-5 writes. But they are new, and the drivers are still maturing.

Yeah...

FWIW there was a thread last month on fedora-devel:

• 3w-9xxx module version in FC4 https://www.redhat.com/archives/fedora-devel-list/2005-April/thread.html#008...

I thought this comment was interesting:

We never use these systems for high usage scenarios like a database server or sometimes even a home directory server. Nearline backup and slow storage is what we consider them useful for.

https://www.redhat.com/archives/fedora-devel-list/2005-April/msg00886.html

I'm in the midst of testing a dual 9500-12 based system, and I've got all sorts of results (I posted tiobench numbers for XFS and ext3 recently). I've been playing with IOR http://www.llnl.gov/asci/purple/benchmarks/limited/ior/, ftp://ftp.llnl.gov/pub/siop/ior/ for the past couple of days. This is the output from a sample run across 10 clients (NFS over tcp, wsize=rsize=32768, centos-3 clients, centos-4 server):

Command line used: /home/jlb/src/IOR-2.8.4/src/C/IOR -F -W -R -b 40m -t 4m -s 103 -e Participating tasks: 10

Summary: api = POSIX test filename = testFile access = file-per-process clients = 10 (1 per node) repetitions = 1 xfersize = 4 MiB blocksize = 40 MiB aggregate filesize = 40.23 GiB Lustre stripe size = Use default stripe count = Use default

access bw(MiB/s) block(KiB) xfer(KiB) open(s) wr/rd(s) close(s) iter ------ --------- ---------- --------- -------- -------- -------- ---- write 25.92 40960 4096 0.067203 1589.32 78.92 0 read 208.72 40960 4096 0.093569 197.39 191.23 0

Note that the server is dual homed, with half of the clients accessing each address -- thus the read number (yes, gigabit everywhere). For that test, the server was running XFS. Doing the same test with ext3 the write number is slightly higher (~30 MiB/s) and the read number slightly lower (~190MiB/s).

Just putting it out there.

On Wed, 11 May 2005 at 8:58pm, Lee W wrote

Thanks for all the advise people have offered. I've been checking out all the 3ware cards available and believe that the Escalade 7506-4LP would be best for me.

However I've noticed that it is a 64-Bit PCI Card. Can I still use this in a 32-Bit Slot. I think that I read something once that said it would work but at a reduced bandwidth.

Page 16 of the Installation Guide says a 32bit slot is fine.

I use one of my 8500-8s in a 32bit slot on an MSI Dual PIII motherboard with no problems... You just have to make sure the 64bit part of the connector does not foul on any motherboard parts as it hangs over the edge of the 32bit slot...

Regards

Pete

Joshua Baker-LePain wrote:

On Wed, 11 May 2005 at 8:58pm, Lee W wrote

...

Thanks for all the advise people have offered. I've been checking out all the 3ware cards available and believe that the Escalade 7506-4LP would be best for me.

However I've noticed that it is a 64-Bit PCI Card. Can I still use this in a 32-Bit Slot. I think that I read something once that said it would work but at a reduced bandwidth.

Page 16 of the Installation Guide says a 32bit slot is fine.