[CentOS] 3Ware 9550SX-4LP Performance
Kirk Bocek
t004 at kbocek.com
Sun Oct 1 05:49:23 UTC 2006
Thanks for the reply, Joshua. Looks like you're right about 3Ware recommending
that NCQ be disabled. The settings listed at
http://www.3ware.com/KB/article.aspx?id=14952 show NCQ disabled. More
importantly, it shows that I *should* be able to get 185MB/Sec writes with
four disks. http://www.3ware.com/kb/article.aspx?id=11050 also mentions making
the blockdev adjustment.
First off, I just tried XFS without changing anything else. I downloaded the
XFS kernel modules and the XFS tools from centosplus, created a 100GB logical
volume and formatted it with xfs. The results:
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Beryl 10G:64k 56595 91 132091 23 42049 6 45990 72 153534 13 195.7 1
Beryl 10G:64k 60359 94 127813 24 42298 6 48124 73 162938 14 191.0 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
Beryl 16 1990 9 +++++ +++ 3019 14 1880 9 +++++ +++ 643 3
Beryl 16 1802 8 +++++ +++ 2776 12 1696 8 +++++ +++ 676 3
Whoa! Block writes jumped to 130MB/Sec with no other changes. XFS Rocks! Next,
I disabled NCQ and set set StorSave to "Balance":
------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
Beryl 10G:64k 58142 93 202483 35 55925 8 46244 73 160890 14 200.6 2
Beryl 10G:64k 60640 94 204090 38 55043 8 48603 74 168764 15 200.8 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
Beryl 16 3112 15 +++++ +++ 4276 19 2673 13 +++++ +++ 811 4
Beryl 16 3242 16 +++++ +++ 4249 19 2903 15 +++++ +++ 828 4
Double Whoa! Block writes increased to 202MB/Sec. Very acceptable. I'll play
with blockdev next and see if I can increase the read speeds.
Thanks again for your suggestions.
Kirk Bocek
Joshua Baker-LePain wrote:
> On Sat, 30 Sep 2006 at 8:56pm, Kirk Bocek wrote
>
>> 3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI
>> Express version of this card (the 9580 I think.) I'm guessing 4x PCI
>> Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The
>> only two things I need to change are to put the disks in 3GB/Sec SATA
>> mode (they're in 1.5GB/Sec mode right now) and to enable memory
>> interleaving on the motherboard (I only have two sticks of memory,
>> this M/B needs 4 for interleaving.) But I don't think these things
>> should drop me down to the write performance of a single IDE disk.
>>
>> Does anyone have any suggestions on improving write performance?
>
> 1) Make sure the firmware on the board is up to date.
>
> 2) Turn off NCQ (last I knew, this was 3ware's advice).
>
> 3) Try putting the ext3 journal on another device.
>
> 4) Use XFS.
>
> The last one is not just a prejudice. There really seems to be some
> sort of odd (and rather bad) interaction between 3ware RAID5 and ext3.
> Going to XFS really helps (especially with writes).
>
> Oh, and be sure to 'blockdev --setra 16384' that device, which will help
> with reads. 16384 is what works for me -- YMMV, of course.
>
More information about the CentOS
mailing list