I know there are a few 3Ware fans here and I was hoping to find some help. I just built a new server using a 3Ware 9550SX-4LP with four disks in raid 5. The array is fully initialized but I'm not getting the write performance I was hoping for -- only 40 to 45MB/Sec.
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode right now) and to enable memory interleaving on the motherboard (I only have two sticks of memory, this M/B needs 4 for interleaving.) But I don't think these things should drop me down to the write performance of a single IDE disk.
Does anyone have any suggestions on improving write performance?
The Details (I know the devil is in here somewhere):
Supermicro X7DVL-E Motherboard Dual Xeon 5130 (2 Ghz) 2GB RAM 3Ware 9550SX-4LP in a 133Mhz 64-bit slot. 4x Seagate ST3400620AS 400GB SATA Drives NCQ Enabled but currently in 1.5GB/Sec mode CentOS 4.4, Kernel 2.6.9-42.0.2.ELsmp Stock Kernel 3Ware Driver 2.26.04.010
Bonnie++ 1.03a 50GB Benchmark run four times on an ext3 filesystem:
------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 50G:64k 44572 77 53084 14 40624 7 47790 72 162504 12 96.5 0 Beryl 50G:64k 45960 79 53068 14 39972 7 47701 73 162510 12 96.8 0 Beryl 50G:64k 36846 64 37579 10 36785 6 48090 72 162609 12 96.9 0 Beryl 50G:64k 45205 77 53621 15 40757 7 48325 73 162331 12 96.4 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 3928 99 +++++ +++ +++++ +++ 3961 98 +++++ +++ 12442 99 Beryl 16 3929 98 +++++ +++ +++++ +++ 3957 99 +++++ +++ 12373 100 Beryl 16 3718 94 +++++ +++ +++++ +++ 3389 84 +++++ +++ 12307 99 Beryl 16 3920 98 +++++ +++ +++++ +++ 3969 99 +++++ +++ 12421 99
On Sat, 30 Sep 2006 at 8:56pm, Kirk Bocek wrote
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode right now) and to enable memory interleaving on the motherboard (I only have two sticks of memory, this M/B needs 4 for interleaving.) But I don't think these things should drop me down to the write performance of a single IDE disk.
Does anyone have any suggestions on improving write performance?
1) Make sure the firmware on the board is up to date.
2) Turn off NCQ (last I knew, this was 3ware's advice).
3) Try putting the ext3 journal on another device.
4) Use XFS.
The last one is not just a prejudice. There really seems to be some sort of odd (and rather bad) interaction between 3ware RAID5 and ext3. Going to XFS really helps (especially with writes).
Oh, and be sure to 'blockdev --setra 16384' that device, which will help with reads. 16384 is what works for me -- YMMV, of course.
Thanks for the reply, Joshua. Looks like you're right about 3Ware recommending that NCQ be disabled. The settings listed at http://www.3ware.com/KB/article.aspx?id=14952 show NCQ disabled. More importantly, it shows that I *should* be able to get 185MB/Sec writes with four disks. http://www.3ware.com/kb/article.aspx?id=11050 also mentions making the blockdev adjustment.
First off, I just tried XFS without changing anything else. I downloaded the XFS kernel modules and the XFS tools from centosplus, created a 100GB logical volume and formatted it with xfs. The results:
------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 10G:64k 56595 91 132091 23 42049 6 45990 72 153534 13 195.7 1 Beryl 10G:64k 60359 94 127813 24 42298 6 48124 73 162938 14 191.0 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 1990 9 +++++ +++ 3019 14 1880 9 +++++ +++ 643 3 Beryl 16 1802 8 +++++ +++ 2776 12 1696 8 +++++ +++ 676 3
Whoa! Block writes jumped to 130MB/Sec with no other changes. XFS Rocks! Next, I disabled NCQ and set set StorSave to "Balance":
------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 10G:64k 58142 93 202483 35 55925 8 46244 73 160890 14 200.6 2 Beryl 10G:64k 60640 94 204090 38 55043 8 48603 74 168764 15 200.8 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 3112 15 +++++ +++ 4276 19 2673 13 +++++ +++ 811 4 Beryl 16 3242 16 +++++ +++ 4249 19 2903 15 +++++ +++ 828 4
Double Whoa! Block writes increased to 202MB/Sec. Very acceptable. I'll play with blockdev next and see if I can increase the read speeds.
Thanks again for your suggestions.
Kirk Bocek
Joshua Baker-LePain wrote:
On Sat, 30 Sep 2006 at 8:56pm, Kirk Bocek wrote
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode right now) and to enable memory interleaving on the motherboard (I only have two sticks of memory, this M/B needs 4 for interleaving.) But I don't think these things should drop me down to the write performance of a single IDE disk.
Does anyone have any suggestions on improving write performance?
Make sure the firmware on the board is up to date.
Turn off NCQ (last I knew, this was 3ware's advice).
Try putting the ext3 journal on another device.
Use XFS.
The last one is not just a prejudice. There really seems to be some sort of odd (and rather bad) interaction between 3ware RAID5 and ext3. Going to XFS really helps (especially with writes).
Oh, and be sure to 'blockdev --setra 16384' that device, which will help with reads. 16384 is what works for me -- YMMV, of course.
On 9/30/06, Kirk Bocek t004@kbocek.com wrote:
Whoa! Block writes jumped to 130MB/Sec with no other changes. XFS Rocks!
Too bad it's not supported and QA'd by the upstream vendor and therefore shouldn't be used in production on CentOS :-(
Francois Caen wrote:
On 9/30/06, Kirk Bocek t004@kbocek.com wrote:
Whoa! Block writes jumped to 130MB/Sec with no other changes. XFS Rocks!
Too bad it's not supported and QA'd by the upstream vendor and therefore shouldn't be used in production on CentOS :-(
Well hey! I think the downstream vendor rocks too! And it sounds like centosplus has done some defacto QA too. :)
Doing some further benchmarking, I added 'blockdev --setra 32768 /dev/sdb' to my previous setup and ran a bunch of benchmarks overnight. The results:
------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 50G:256k 58911 93 198971 36 57194 8 47548 73 166183 14 62.2 2 Beryl 50G:256k 60773 94 196980 36 56325 8 47719 74 166580 14 61.8 2 Beryl 50G:256k 59977 94 187017 36 55981 8 47975 74 165562 14 64.0 2 Beryl 50G:256k 60193 94 196472 36 56896 8 47913 74 166449 14 63.0 2 Beryl 50G:256k 59802 94 201524 38 56746 8 48084 75 168710 14 64.7 1 Beryl 50G:256k 60677 94 199188 37 52997 7 47682 74 163131 15 64.0 2 Beryl 50G:256k 60061 94 191342 37 55949 9 47794 74 165348 15 64.3 2 Beryl 50G:256k 60261 94 184231 35 56235 8 48090 74 166860 14 61.8 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 3597 17 +++++ +++ 4088 18 3101 15 +++++ +++ 856 4 Beryl 16 3240 16 +++++ +++ 4561 20 3030 15 +++++ +++ 755 3 Beryl 16 3039 14 +++++ +++ 4446 20 3111 15 +++++ +++ 754 3 Beryl 16 3317 16 +++++ +++ 4509 21 2978 15 +++++ +++ 767 3 Beryl 16 5037 25 +++++ +++ 4801 20 3349 17 +++++ +++ 891 4 Beryl 16 3390 16 +++++ +++ 3341 14 3222 17 +++++ +++ 806 3 Beryl 16 3547 17 +++++ +++ 3700 16 2986 15 +++++ +++ 799 4 Beryl 16 3556 18 +++++ +++ 3183 13 3391 17 +++++ +++ 863 4
Block writes are still close to 200MB/Sec but block reads still haven't changed from 166MB/Sec. :(
Anyone have any thoughts on increasing read speed other than using blockdev?
Kirk Bocek
On Sun, 2006-10-01 at 08:09 -0700, Kirk Bocek wrote:
Francois Caen wrote:
On 9/30/06, Kirk Bocek t004@kbocek.com wrote:
Whoa! Block writes jumped to 130MB/Sec with no other changes. XFS Rocks!
Too bad it's not supported and QA'd by the upstream vendor and therefore shouldn't be used in production on CentOS :-(
Well hey! I think the downstream vendor rocks too! And it sounds like centosplus has done some defacto QA too. :)
Doing some further benchmarking, I added 'blockdev --setra 32768 /dev/sdb' to my previous setup and ran a bunch of benchmarks overnight. The results:
------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 50G:256k 58911 93 198971 36 57194 8 47548 73 166183 14 62.2 2 Beryl 50G:256k 60773 94 196980 36 56325 8 47719 74 166580 14 61.8 2 Beryl 50G:256k 59977 94 187017 36 55981 8 47975 74 165562 14 64.0 2 Beryl 50G:256k 60193 94 196472 36 56896 8 47913 74 166449 14 63.0 2 Beryl 50G:256k 59802 94 201524 38 56746 8 48084 75 168710 14 64.7 1 Beryl 50G:256k 60677 94 199188 37 52997 7 47682 74 163131 15 64.0 2 Beryl 50G:256k 60061 94 191342 37 55949 9 47794 74 165348 15 64.3 2 Beryl 50G:256k 60261 94 184231 35 56235 8 48090 74 166860 14 61.8 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 3597 17 +++++ +++ 4088 18 3101 15 +++++ +++ 856 4 Beryl 16 3240 16 +++++ +++ 4561 20 3030 15 +++++ +++ 755 3 Beryl 16 3039 14 +++++ +++ 4446 20 3111 15 +++++ +++ 754 3 Beryl 16 3317 16 +++++ +++ 4509 21 2978 15 +++++ +++ 767 3 Beryl 16 5037 25 +++++ +++ 4801 20 3349 17 +++++ +++ 891 4 Beryl 16 3390 16 +++++ +++ 3341 14 3222 17 +++++ +++ 806 3 Beryl 16 3547 17 +++++ +++ 3700 16 2986 15 +++++ +++ 799 4 Beryl 16 3556 18 +++++ +++ 3183 13 3391 17 +++++ +++ 863 4
Block writes are still close to 200MB/Sec but block reads still haven't changed from 166MB/Sec. :(
Anyone have any thoughts on increasing read speed other than using blockdev?
If you are using XFS ... make sure to use the standalone modules and not the versions with XFS compiled into the kernel.
The standalone modules are much newer code from SGI.
I would also recommend using the x86_64 arch if it can be used as the x86_64 kernel is more stable w/XFS due to the 4K stacks issue that sometimes causes problems with the i386 arch.
Johnny Hughes wrote:
If you are using XFS ... make sure to use the standalone modules and not the versions with XFS compiled into the kernel.
The standalone modules are much newer code from SGI.
I would also recommend using the x86_64 arch if it can be used as the x86_64 kernel is more stable w/XFS due to the 4K stacks issue that sometimes causes problems with the i386 arch.
Oooh, I hit a subject close to the heart of one of the developers! :)
I installed the centosplus kernel so I could play with JFS and ReiserFS as well as XFS. However, I don't think the XFS modules are present in the centosplus kernel package. At least I didn't find them under /lib/modules until after installing the separate kernel-module-xfs packages.
Already running x86_64 -- this host has dual Xeon 5130s.
So here's the big question, what are the pluses and minuses of XFS vs JFS vs ReiserFS? Below are some quick benchmarks. XFS clearly wins the speed contest unless there's some tuning that can be done. Any thoughts?
Kirk Bocek
XFS: ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 10G:64k 58142 93 202483 35 55925 8 46244 73 160890 14 200.6 2 Beryl 10G:64k 60640 94 204090 38 55043 8 48603 74 168764 15 200.8 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 3112 15 +++++ +++ 4276 19 2673 13 +++++ +++ 811 4 Beryl 16 3242 16 +++++ +++ 4249 19 2903 15 +++++ +++ 828 4
JFS: ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 10G:64k 58559 91 152961 22 61014 8 46598 75 167160 15 204.6 2 Beryl 10G:64k 58952 92 153184 22 60521 8 49865 74 173444 15 200.3 1 Beryl 10G:64k 58942 92 148911 22 60246 8 49980 75 172518 15 193.4 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 8829 14 +++++ +++ 10052 19 6571 31 +++++ +++ 3123 9 Beryl 16 7483 14 +++++ +++ 10110 20 6329 28 +++++ +++ 1257 4 Beryl 16 9601 18 +++++ +++ 9890 19 6438 28 +++++ +++ 1816 5
ReiserFS: ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 10G:64k 59667 93 83212 18 37790 6 41001 67 108422 13 168.1 1 Beryl 10G:64k 59390 93 83568 18 41905 7 46799 71 110146 13 168.4 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 +++++ +++ +++++ +++ 27965 100 32467 100 +++++ +++ 25871 100 Beryl 16 +++++ +++ +++++ +++ 27676 99 +++++ +++ +++++ +++ 26409 99
I know there are a few 3Ware fans here and I was hoping to find some help. I just built a new server using a 3Ware 9550SX-4LP with four disks in raid 5. The array is fully initialized but I'm not getting the write performance I was hoping for -- only 40 to 45MB/Sec.
I've got similar trouble on old Xeon MBs with 5 PCI-X slot. Try to use the other slot and clear PCI information on BIOS.
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode right now) and to enable memory interleaving on the motherboard (I only have two sticks of memory, this M/B needs 4 for interleaving.) But I don't think these things should drop me down to the write performance of a single IDE disk.
Does anyone have any suggestions on improving write performance?
The Details (I know the devil is in here somewhere):
Supermicro X7DVL-E Motherboard Dual Xeon 5130 (2 Ghz) 2GB RAM 3Ware 9550SX-4LP in a 133Mhz 64-bit slot. 4x Seagate ST3400620AS 400GB SATA Drives NCQ Enabled but currently in 1.5GB/Sec mode CentOS 4.4, Kernel 2.6.9-42.0.2.ELsmp Stock Kernel 3Ware Driver 2.26.04.010
Bonnie++ 1.03a 50GB Benchmark run four times on an ext3 filesystem:
------Sequential Output------ --Sequential Input---Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP Beryl 50G:64k 44572 77 53084 14 40624 7 47790 72 162504 12 96.5 0 Beryl 50G:64k 45960 79 53068 14 39972 7 47701 73 162510 12 96.8 0 Beryl 50G:64k 36846 64 37579 10 36785 6 48090 72 162609 12 96.9 0 Beryl 50G:64k 45205 77 53621 15 40757 7 48325 73 162331 12 96.4 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files:max:min /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP Beryl 16 3928 99 +++++ +++ +++++ +++ 3961 98 +++++ +++ 12442 99 Beryl 16 3929 98 +++++ +++ +++++ +++ 3957 99 +++++ +++ 12373 100 Beryl 16 3718 94 +++++ +++ +++++ +++ 3389 84 +++++ +++ 12307 99 Beryl 16 3920 98 +++++ +++ +++++ +++ 3969 99 +++++ +++ 12421 99
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Sat, 2006-09-30 at 20:56 -0700, Kirk Bocek wrote:
I know there are a few 3Ware fans here and I was hoping to find some help. I just built a new server using a 3Ware 9550SX-4LP with four disks in raid 5. The array is fully initialized but I'm not getting the write performance I was hoping for -- only 40 to 45MB/Sec.
If you are satisfied with the data integrity guarantees of XFS, which are weaker than what EXT3 gives you by default, you might want to mount your EXT3 filesystem with the data=writeback option.
This turns off the extra guarantees that ext3 gives you.
And if you do use XFS, make absolutely certain you can trust your UPS and that the system is configured to do a normal shutdown before the UPS runs out.
I've heard too many horror stories about XFS filesystems destroyed by a power blip.
Yes, I'm aware that it says in their FAQ that this was a problem that was fixed in 1.1. But from what I'm hearing, it is perhaps better, but not fixed.
Just to compare the guarantees:
Ext3 (data=journal): After an unclean shutdown, Fileysystem and metadata are guaranteed to be consistent. If the application was told that a data write occurred, it did. (Slowest writes.)
Ext3 (data=ordered): After a power outage, the filesystem and metadata are guaranteed to be consistent. The file contents are guaranteed not to be garbage. BUT, the data might not be the latest written. (Medium speed writes.)
Ext3 (data=writeback): This is the level that most journaled filesystems give you. After an unclean shutdown, filesystem and metadata are guaranteed to be consistent. File can contain garbage. (Fastest writes.)
XFS: Same as data=writeback for ext3, but due to the way XFS nulls out data blocks, and also delays allocation, data loss is more likely.
Someone more familiar with XFS might want to elaborate or correct my sketchy description of the reasons for XFS's relative fragility.
Be advised that changing the data journaling option for a *root* filesystem is more involved than just changing it in /etc/fstab. For /home, /var, etc. you can just add it to fstab and it will work.
-Steve
Kirk Bocek wrote:
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode right now) and to enable memory interleaving on the motherboard (I only have two sticks of memory, this M/B needs 4 for interleaving.) But I don't think these things should drop me down to the write performance of a single IDE disk.
Has somebody made experience with PCI-X-133MHz and PCI-X-66MHz? With my board there's not much difference. I even have the feeling, that driving a 9550SXU-4LP at PCI-X-66MHz is a bit faster than at 133. At least bonnie++ gives me slightly faster values @ 66.
Does anyone have any suggestions on improving write performance?
see above. :)
TIA - Michael
The mobo I'm using in this setup allows downshifting PCI-X slots to 100 and 66 Mhz. Alas, it's in production now and I'm not able to test those configurations.
Kirk Bocek
Michael Kress wrote:
Kirk Bocek wrote:
3Ware's site advertises 300MB/Sec writes using 8 disks on the PCI Express version of this card (the 9580 I think.) I'm guessing 4x PCI Express has more bandwidth but 133Mhz 64-bit PCI-X is no slouch. The only two things I need to change are to put the disks in 3GB/Sec SATA mode (they're in 1.5GB/Sec mode right now) and to enable memory interleaving on the motherboard (I only have two sticks of memory, this M/B needs 4 for interleaving.) But I don't think these things should drop me down to the write performance of a single IDE disk.
Has somebody made experience with PCI-X-133MHz and PCI-X-66MHz? With my board there's not much difference. I even have the feeling, that driving a 9550SXU-4LP at PCI-X-66MHz is a bit faster than at 133. At least bonnie++ gives me slightly faster values @ 66.
Does anyone have any suggestions on improving write performance?
see above. :)
TIA - Michael
-
Francois Caen -
Johnny Hughes -
Joshua Baker-LePain -
Kirk Bocek -
Michael Kress -
Morimoto Kenji -
Steve Bergman