Thanks for the reply, Joshua. Looks like you're right about 3Ware recommending that NCQ be disabled. The settings listed at http://www.3ware.com/KB/article.aspx?id=14952 show NCQ disabled. More importantly, it shows that I *should* be able to get 185MB/Sec writes with four disks. http://www.3ware.com/kb/article.aspx?id=11050 also mentions making the blockdev adjustment.
First off, I just tried XFS without changing anything else. I downloaded the XFS kernel modules and the XFS tools from centosplus, created a 100GB logical volume and formatted it with xfs. The results:
------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 10G:64k 56595 91 132091 23 42049 6 45990 72 153534 13 195.7 1 Beryl 10G:64k 60359 94 127813 24 42298 6 48124 73 162938 14 191.0 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 1990 9 +++++ +++ 3019 14 1880 9 +++++ +++ 643 3 Beryl 16 1802 8 +++++ +++ 2776 12 1696 8 +++++ +++ 676 3
Whoa! Block writes jumped to 130MB/Sec with no other changes. XFS Rocks! Next, I disabled NCQ and set set StorSave to "Balance":
------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 10G:64k 58142 93 202483 35 55925 8 46244 73 160890 14 200.6 2 Beryl 10G:64k 60640 94 204090 38 55043 8 48603 74 168764 15 200.8 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 3112 15 +++++ +++ 4276 19 2673 13 +++++ +++ 811 4 Beryl 16 3242 16 +++++ +++ 4249 19 2903 15 +++++ +++ 828 4
Double Whoa! Block writes increased to 202MB/Sec. Very acceptable. I'll play with blockdev next and see if I can increase the read speeds.
Thanks again for your suggestions.
Kirk Bocek
Joshua Baker-LePain wrote:
On Sat, 30 Sep 2006 at 8:56pm, Kirk Bocek wrote
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode right now) and to enable memory interleaving on the motherboard (I only have two sticks of memory, this M/B needs 4 for interleaving.) But I don't think these things should drop me down to the write performance of a single IDE disk.
Does anyone have any suggestions on improving write performance?
Make sure the firmware on the board is up to date.
Turn off NCQ (last I knew, this was 3ware's advice).
Try putting the ext3 journal on another device.
Use XFS.
The last one is not just a prejudice. There really seems to be some sort of odd (and rather bad) interaction between 3ware RAID5 and ext3. Going to XFS really helps (especially with writes).
Oh, and be sure to 'blockdev --setra 16384' that device, which will help with reads. 16384 is what works for me -- YMMV, of course.