Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
dd if=/dev/zero of=/logs/test bs=100M count=1000
(100GB of data, written in 100MB blocks)
It's a Supermicro server, the motherboard is (according to the stuff painted on the mobo itself) Super X7DBI+, Intel Xeon L5420 with 8 cores total, 3ware 9650SE using the driver from the manufacturer's site. I tested 2 disks in a mirror RAID volume.
Results:
kernel time speed ====== ==== ===== patched 75min 23M/s original 51min 34M/s patched 48min 36M/s original 61min 28M/s
On Thu, Jun 19, 2008 at 10:55 AM, Florin Andrei florin@andrei.myip.org wrote:
Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
That kernel (kernel-2.6.18-53.1.21.el5.bz321111) has nothing to do with the issue you are referring to. It was used by Jim as an *example* of what CentOS can offer...
Akemi
On Thu, 19 Jun 2008 at 10:55am, Florin Andrei wrote
Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
That's the wrong patched kernel. You'd need to be using one of the kernels in http://people.redhat.com/thenzl/kernel/ -- kernel-2.6.18-93.el5.bz444759.x86_64.rpm.
I'd be interested in a way of telling from within the OS whether or not MWI is enabled...
Joshua Baker-LePain wrote:
That's the wrong patched kernel. You'd need to be using one of the kernels in http://people.redhat.com/thenzl/kernel/ -- kernel-2.6.18-93.el5.bz444759.x86_64.rpm.
(facepalm) Thanks for setting me right.
Anyway, I did a test with the 2.6.18-93.el5.bz444759 kernel and there's no difference: 65 minutes, 27 MB/s. Looks like it doesn't matter which kernel I use, at least for this simple test with dd.
I wonder if a test closer to real life, such as reading/writing stuff from/to MySQL, would produce different results. I guess there's only one way to find out. ;-)
I'd be interested in a way of telling from within the OS whether or not MWI is enabled...
That would be nice.
Florin Andrei wrote:
Anyway, I did a test with the 2.6.18-93.el5.bz444759 kernel and there's no difference: 65 minutes, 27 MB/s. Looks like it doesn't matter which kernel I use, at least for this simple test with dd.
I wonder if a test closer to real life, such as reading/writing stuff from/to MySQL, would produce different results. I guess there's only one way to find out. ;-)
As a side note, the artificial benchmark reveals a huge difference between Ext3 and XFS - the latter is much faster when writing. Might be an artifact of some setting (after all, I do use a hardware RAID card). But the difference is very real.
I was planning to use XFS anyway, so I'm not sure if I'll spend too much time troubleshooting Ext3.
I don't think this is some kind of hidden effect of the MWI bug.
On Fri, 20 Jun 2008 at 9:55am, Florin Andrei wrote
Florin Andrei wrote:
Anyway, I did a test with the 2.6.18-93.el5.bz444759 kernel and there's no difference: 65 minutes, 27 MB/s. Looks like it doesn't matter which kernel I use, at least for this simple test with dd.
I wonder if a test closer to real life, such as reading/writing stuff from/to MySQL, would produce different results. I guess there's only one way to find out. ;-)
As a side note, the artificial benchmark reveals a huge difference between Ext3 and XFS - the latter is much faster when writing. Might be an artifact of some setting (after all, I do use a hardware RAID card). But the difference is very real.
I was planning to use XFS anyway, so I'm not sure if I'll spend too much time troubleshooting Ext3.
I don't think this is some kind of hidden effect of the MWI bug.
XFS has *always* been faster on 3ware than ext3. RH has never been interested in looking at why. *shrug*
On Fri, Jun 20, 2008 at 12:55 PM, Florin Andrei florin@andrei.myip.org wrote:
Anyway, I did a test with the 2.6.18-93.el5.bz444759 kernel and there's no difference: 65 minutes, 27 MB/s. Looks like it doesn't matter which kernel I use, at least for this simple test with dd.
No. Since you're using the driver from 3ware/amcc you are not going to experience this particular performance hit. Only if you're using the stock 3ware driver which is built into the kernel.
As a side note, the artificial benchmark reveals a huge difference between Ext3 and XFS - the latter is much faster when writing. Might be an artifact of some setting (after all, I do use a hardware RAID card). But the difference is very real.
I was planning to use XFS anyway, so I'm not sure if I'll spend too much time troubleshooting Ext3.
I don't think this is some kind of hidden effect of the MWI bug.
Nope. This is just a long-standing performance thing. You can tune ext3 to perform better, but on a 3ware card xfs will win, hands down.
Jim Perrin wrote:
Nope. This is just a long-standing performance thing. You can tune ext3 to perform better, but on a 3ware card xfs will win, hands down.
of course, XFS can also fail spectacularly. ext3fs fully journals all metadata updates. I'm sure this is a major portion of the performance differences on writes.
On Fri, 20 Jun 2008 at 10:34am, John R Pierce wrote
Jim Perrin wrote:
Nope. This is just a long-standing performance thing. You can tune ext3 to perform better, but on a 3ware card xfs will win, hands down.
of course, XFS can also fail spectacularly. ext3fs fully journals all metadata updates. I'm sure this is a major portion of the performance differences on writes.
Every FS can fail spectacularly. XFS (obviosly) journals as well, but it doesn't force an "ordered" mode as ext3 does by default. However, even if you mount ext3 with "data=writeback" (which is roughly analogous to XFS' journaling mode), ext3 still doesn't perform nearly as well as XFS on 3ware.
John R Pierce wrote:
of course, XFS can also fail spectacularly. ext3fs fully journals all metadata updates. I'm sure this is a major portion of the performance differences on writes.
Actually, I've used XFS since the days it was released as a port to Linux (and even before that, on Irix, but that's besides the point). I'm aware it's more fragile than Ext3 - in fact, someone here at the office made an XFS partition yesterday to do some tests, there was a power outage last night, and today that partition is corrupted.
I'll use battery backups (duh) and only put on XFS the stuff that needs good performance, but can be rebuilt from the master data in case something ugly happens. Like pretty much anything in life, it's a trade-off.
Florin Andrei wrote:
John R Pierce wrote:
of course, XFS can also fail spectacularly. ext3fs fully journals all metadata updates. I'm sure this is a major portion of the performance differences on writes.
Actually, I've used XFS since the days it was released as a port to Linux (and even before that, on Irix, but that's besides the point). I'm aware it's more fragile than Ext3 - in fact, someone here at the office made an XFS partition yesterday to do some tests, there was a power outage last night, and today that partition is corrupted.
I'll use battery backups (duh) and only put on XFS the stuff that needs good performance, but can be rebuilt from the master data in case something ugly happens. Like pretty much anything in life, it's a trade-off.
Also to make sure to only use kernel drivers that are solid under both normal and heavy loads, burn in the hardware sufficiently to make sure there are no lurking failures and make sure your applications behave properly.
It's not just power that can stop a server suddenly, but to XFS the result is the same!
-Ross
______________________________________________________________________ This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this e-mail in error, please immediately notify the sender and permanently delete the original and any copy or printout thereof.
Jim Perrin wrote:
Nope. This is just a long-standing performance thing. You can tune ext3 to perform better, but on a 3ware card xfs will win, hands down.
Well, after performing more mental ruminations on this subject, restoring the database in case of a crash might be a pain, so I may have to avoid XFS in this particular case.
Can somebody recommend a RAID card, about the same price range like the 3ware 9650SE, and with similar features, 8 SATA ports or more, that works well with CentOS 5 and does not have the Ext3 performance bug?
There's a bunch of cards out there, but I'd like to hear from someone with first-hand experience with one of them.
On Thu, Jun 19, 2008 at 1:55 PM, Florin Andrei florin@andrei.myip.org wrote:
Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
You are comparing the wrong kernels. The bz321111 kernel was for an NFS related bug and has no 3 impact on 3ware performance. The redhat bug report has the kernels built for 3ware performance. Once 5.2 is rolled out, Johnny Hughes has stated he would consider building a kernel for the 444759 bug.
Jim Perrin wrote:
On Thu, Jun 19, 2008 at 1:55 PM, Florin Andrei florin@andrei.myip.org wrote:
Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
You are comparing the wrong kernels. The bz321111 kernel was for an NFS related bug and has no 3 impact on 3ware performance. The redhat bug report has the kernels built for 3ware performance. Once 5.2 is rolled out, Johnny Hughes has stated he would consider building a kernel for the 444759 bug.
How would I know if I'm affected by this bug? I'm using the 9690SA controller.
Russ
On Thu, Jun 19, 2008 at 4:05 PM, Ruslan Sivak russ@vshift.com wrote:
How would I know if I'm affected by this bug? I'm using the 9690SA controller. Russ
Mostly, if you're using the stock 3w9xxx module that comes in centos, you're affected.
If you're using the 3ware driver from 3ware/amcc, you can look in the source for mwi and see if it's being set/enabled.
on 6-19-2008 1:05 PM Ruslan Sivak spake the following:
Jim Perrin wrote:
On Thu, Jun 19, 2008 at 1:55 PM, Florin Andrei florin@andrei.myip.org wrote:
Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
You are comparing the wrong kernels. The bz321111 kernel was for an NFS related bug and has no 3 impact on 3ware performance. The redhat bug report has the kernels built for 3ware performance. Once 5.2 is rolled out, Johnny Hughes has stated he would consider building a kernel for the 444759 bug.
How would I know if I'm affected by this bug? I'm using the 9690SA controller. Russ
Does the stock drivers even support that controller yet?
Scott Silva wrote:
on 6-19-2008 1:05 PM Ruslan Sivak spake the following:
Jim Perrin wrote:
On Thu, Jun 19, 2008 at 1:55 PM, Florin Andrei florin@andrei.myip.org wrote:
Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
You are comparing the wrong kernels. The bz321111 kernel was for an NFS related bug and has no 3 impact on 3ware performance. The redhat bug report has the kernels built for 3ware performance. Once 5.2 is rolled out, Johnny Hughes has stated he would consider building a kernel for the 444759 bug.
How would I know if I'm affected by this bug? I'm using the 9690SA controller. Russ
Does the stock drivers even support that controller yet?
Not until 5.2
Russ
Jim Perrin wrote:
On Thu, Jun 19, 2008 at 1:55 PM, Florin Andrei florin@andrei.myip.org wrote:
Have a look at these pages:
http://www.bofh-hunter.com/2008/06/13/3ware-performance-in-centos/ https://bugzilla.redhat.com/show_bug.cgi?id=444759
I'm comparing the default 5.1 64bit kernel with the patched one posted in the bug report (kernel-2.6.18-53.1.21.el5.bz321111.x86_64) and I don't quite see any significant difference in write performance for this command:
You are comparing the wrong kernels. The bz321111 kernel was for an NFS related bug and has no 3 impact on 3ware performance. The redhat bug report has the kernels built for 3ware performance. Once 5.2 is rolled out, Johnny Hughes has stated he would consider building a kernel for the 444759 bug.
OK, there is a kernel for this in the Testing repo , the version is:
kernel-2.6.18-92.1.1.el5.bz444759
This kernel is compiled on the 5.2 tree (which should be released before Tuesday 6/24). It might also work OK on CentOS-5.1 but I have not tested it on 5.1
http://dev.centos.org/centos/5/testing/
Thanks, Johnny Hughes
-
Akemi Yagi -
Florin Andrei -
Jim Perrin -
John R Pierce -
Johnny Hughes -
Joshua Baker-LePain -
Ross S. W. Walker -
Ruslan Sivak -
Scott Silva