Hello,
Does anyone have experience using a 3ware 9650SE series raid controller on CentOS 6.0?
I am getting very sporadic throughput with moderately sized files (0.5-2GB) on ext3. I have tried most of the mount time tuning options:
* noatime * trying different journal types * setting commit=120 - helped a little
Even after these optimizations it doesn't seem like the raid array is working as it should. After a few 1GB writes (dd from /dev/zero to the raid) kjournald runs for an hour and later writes are really slow. Not only that, using tw_cli is very slow ... whereas tw_cli is superfast if kjournald is not churning away.
So it goes something like this
* write one 1GB file (486 MB/s) * writes another (223 MB/s) * any writes before the 120s commit kicks in, is ~200MB/s * commit kicks in and kjournald starts churning * writes are all over the map - 6-85MB/s
Maybe this is just the way it is, but it did not seem to be the case this same hardware was running Fedora (Core) 9 and I have a similar machine where this does not seem to be the case (I can't experiment on it at the moment).
The one thing I do not know, since I did not create the RAID or ext3 filesystem is whether the stride and stripe-width were properly selected to match the 64k chunk size of the raid array. I don't know how to tell from tune2fs ... output below ... its either not there or by another name.
Any help or suggestions are appreciated.
Austin
Useful Information ==============
============== /proc/mounts =====================
/dev/sda1 /tonga_raid ext3 rw,noatime,errors=continue,nouser_xattr,noacl,commit=120,barrier=1,data=ordered 0 0
========= tw_cli /c2 show diag =====================
### Time Stamp: 12:34:18 01-Sep-2011 ### Host Name: tonga ### Host Architecture: x86_64 (64 bit) ### OS Version: Linux 2.6.32-71.29.1.el6.x86_64 ### Model: 9650SE-8LPML ### Serial #: L326025A8221043 ### Controller ID: 2 ### CLI Version: 2.00.11.016 ### API Version: 2.08.00.017 ### Driver Version: 2.26.02.014RH ### Firmware Version: FE9X 3.08.00.016 ### BIOS Version: BE9X 3.08.00.004 ### Available Memory: 224MB
========================================================================== Diagnostic Information on Controller //.../c2/... -------------------------------------------------------------------------- Event Trigger and Log Information: Triggered Event(s) = ctlreset (controller soft reset) fwassert (firmware assert) driveerr (drive error) Diagnostic log save mode = - Parameter table does not exist
========== tune2fs -l /dev/sda1 ================= tune2fs 1.41.12 (17-May-2010) Filesystem volume name: /data Last mounted on: <not available> Filesystem UUID: 3c5f6dbb-d5dc-4f85-bc70-9b761c89c86e Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 427245568 Block count: 1708965879 Reserved block count: 85448293 Free blocks: 809305423 Free inodes: 426287552 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 616 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Filesystem created: Tue Sep 9 09:57:44 2008 Last mount time: Thu Sep 1 12:40:01 2011 Last write time: Thu Sep 1 12:40:01 2011 Mount count: 11 Maximum mount count: -1 Last checked: Tue Aug 30 23:12:49 2011 Check interval: 0 (<none>) Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 0b324311-93a9-4c23-bf15-40965792029b Journal backup: inode blocks ===========================================================
=================== tw_cli info c2 ============================
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy ------------------------------------------------------------------------------ u0 RAID-5 OK - - 64K 6519.19 OFF OFF
Port Status Unit Size Blocks Serial --------------------------------------------------------------- p0 OK u0 931.51 GB 1953525168 WD-WCASJ1631953
p1 OK u0 931.51 GB 1953525168 WD-WCASJ1622428
p2 OK u0 931.51 GB 1953525168 WD-WCASJ1639721
p3 OK u0 931.51 GB 1953525168 WD-WCASJ1636054
p4 OK u0 931.51 GB 1953525168 WD-WCASJ1621694
p5 OK u0 931.51 GB 1953525168 WD-WCASJ1636292
p6 OK u0 931.51 GB 1953525168 WD-WCASJ1637586
p7 OK u0 931.51 GB 1953525168 WD-WCASJ1637516
===========================================================
Em 01-09-2011 17:41, Austin Godber escreveu:
=================== tw_cli info c2 ============================
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
u0 RAID-5 OK - - 64K 6519.19 OFF OFF
I have the same controller on Centos 5.
Did you try to active Cache on 3ware?
[17:56:04 root@backup ~]# lspci | grep 3ware 01:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID PCIe (rev 01) [17:56:11 root@backup ~]# tw_cli /c4 show
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy ------------------------------------------------------------------------------ u0 RAID-5 OK - - 64K 5587.9 RiW ON
In the past I had a scenario where I started with Cache OFF and had a poor performance, specially for write. After set Cache RiW I got better performance.
I didn't test that controller on CentOS 6, but is good to know if there is some problem, because my company sells equipment with that controller.
Best regards,
Hello Marcelo,
Thank you for the suggestion. I had not yet tried activating the cache since I was unsure whether that was a good idea or not.
Since you have experience with this card, do you have any recommendations for what I should expect or avoid? Have you used EXT3 with success or are you using XFS or something else?
Austin
On Thu, Sep 1, 2011 at 2:05 PM, Marcelo Beckmann < marcelo.beckmann@webers.com.br> wrote:
Em 01-09-2011 17:41, Austin Godber escreveu:
=================== tw_cli info c2 ============================
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
u0 RAID-5 OK - - 64K 6519.19 OFF OFF
I have the same controller on Centos 5.
Did you try to active Cache on 3ware?
[17:56:04 root@backup ~]# lspci | grep 3ware 01:00.0 RAID bus controller: 3ware Inc 9650SE SATA-II RAID PCIe (rev 01) [17:56:11 root@backup ~]# tw_cli /c4 show
Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy
u0 RAID-5 OK - - 64K 5587.9 RiW ON
In the past I had a scenario where I started with Cache OFF and had a poor performance, specially for write. After set Cache RiW I got better performance.
I didn't test that controller on CentOS 6, but is good to know if there is some problem, because my company sells equipment with that controller.
Best regards,
-- Marcelo Beckmann Suporte Corporativo - suporte@webers.com.br Webers Tecnologia - http://www.webers.com.br Curitiba (PR) (41) 3094-6600 Rio de Janeiro (RJ) (21) 4007-1207 São Paulo (SP) (11) 4007-1207
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
Hello,
Does anyone have experience using a 3ware 9650SE series raid controller on CentOS 6.0?
---- use RAID 10
Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.
Craig
Hi Craig,
Thanks for the suggestion. I would if I could. I'd also probably try another file system. Though the good news is, enabling the write cache on that array has improved things significantly. Which, in my case, was:
tw_cli /c2/u0 set cache=on
Now, if only I had the battery backup unit for the card.
Thanks, everyone for their suggestions. For now I am happy with the situation, but I'd be interested to hear the experiences of others.
Austin
On Thu, Sep 1, 2011 at 3:58 PM, Craig White craig.white@ttiltd.com wrote:
On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
Hello,
Does anyone have experience using a 3ware 9650SE series raid controller
on CentOS 6.0?
use RAID 10
Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.
Craig _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
I'm surprised that you can actually turn it on without a battery. I suspect that this is not a write-through/write-back cache but be forewarned that if there's no battery, it's possible that things you thought were written to the hard drive on shutdown/restart/hang/crash might not ever be written to the hard drive(s)
Craig
On Sep 1, 2011, at 4:17 PM, Austin Godber wrote:
Hi Craig,
Thanks for the suggestion. I would if I could. I'd also probably try another file system. Though the good news is, enabling the write cache on that array has improved things significantly. Which, in my case, was:
tw_cli /c2/u0 set cache=on
Now, if only I had the battery backup unit for the card.
Thanks, everyone for their suggestions. For now I am happy with the situation, but I'd be interested to hear the experiences of others.
Austin
On Thu, Sep 1, 2011 at 3:58 PM, Craig White craig.white@ttiltd.com wrote:
On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
Hello,
Does anyone have experience using a 3ware 9650SE series raid controller on CentOS 6.0?
use RAID 10
Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.
Keep in mind you really only want to enable the cache if you have a bbc, otherwise you are risking your data since it can/will cache writes...just something to keep in mind.
On 9/1/11, Austin Godber godber@gmail.com wrote:
Hi Craig,
Thanks for the suggestion. I would if I could. I'd also probably try another file system. Though the good news is, enabling the write cache on that array has improved things significantly. Which, in my case, was:
tw_cli /c2/u0 set cache=on
Now, if only I had the battery backup unit for the card.
Thanks, everyone for their suggestions. For now I am happy with the situation, but I'd be interested to hear the experiences of others.
Austin
On Thu, Sep 1, 2011 at 3:58 PM, Craig White craig.white@ttiltd.com wrote:
On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
Hello,
Does anyone have experience using a 3ware 9650SE series raid controller
on CentOS 6.0?
use RAID 10
Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.
Craig _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
At this point the card is pretty much useless without that cache enabled. Without recommendations for making writes of 256MB or larger files faster without this cache enabled, I will have to accept the possible data loss in the event of power outage. If it is only the case of data loss during a power outage, I will take that ... rather than failure to write at all during 99% of my usage.
I will, for the sake of not being an idiot, look into buying the BBUs.
Austin
On Thu, Sep 1, 2011 at 4:49 PM, Tom Bishop bishoptf@gmail.com wrote:
Keep in mind you really only want to enable the cache if you have a bbc, otherwise you are risking your data since it can/will cache writes...just something to keep in mind.
On 9/1/11, Austin Godber godber@gmail.com wrote:
Hi Craig,
Thanks for the suggestion. I would if I could. I'd also probably try another file system. Though the good news is, enabling the write cache
on
that array has improved things significantly. Which, in my case, was:
tw_cli /c2/u0 set cache=on
Now, if only I had the battery backup unit for the card.
Thanks, everyone for their suggestions. For now I am happy with the situation, but I'd be interested to hear the experiences of others.
Austin
On Thu, Sep 1, 2011 at 3:58 PM, Craig White craig.white@ttiltd.com
wrote:
On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
Hello,
Does anyone have experience using a 3ware 9650SE series raid
controller
on CentOS 6.0?
use RAID 10
Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.
Craig _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Data loss could conceivably occur on shutdown or restart too - just saying... You are assuming that the data that doesn't get written to disk is going to be non-essential... I wish you good luck with that. I think if one doesn't want to be an idiot, one would not enable a cache that has no means to ensure that the cache is written to disk.
I think your take away from all of this is somewhat misdirected. Not having a BBU simply means that your writes really should always be synchronous/immediate. That shouldn't really be a problem and shouldn't impose a large performance penalty.
Your performance issue relates more to the fact that RAID 5 implementation on the 3Ware cards is rather poor and modes such as RAID 10 (RAID 0 + 1) will give you much more speed that you realize. If you also consider on the surprisingly higher rates of failure with loss of data possibility when reconstructing a missing/dead drive on a RAID 5 setup you really should be re-examining your storage strategy.
Craig
On Sep 1, 2011, at 5:43 PM, Austin Godber wrote:
At this point the card is pretty much useless without that cache enabled. Without recommendations for making writes of 256MB or larger files faster without this cache enabled, I will have to accept the possible data loss in the event of power outage. If it is only the case of data loss during a power outage, I will take that ... rather than failure to write at all during 99% of my usage.
I will, for the sake of not being an idiot, look into buying the BBUs.
Austin
On Thu, Sep 1, 2011 at 4:49 PM, Tom Bishop bishoptf@gmail.com wrote: Keep in mind you really only want to enable the cache if you have a bbc, otherwise you are risking your data since it can/will cache writes...just something to keep in mind.
On 9/1/11, Austin Godber godber@gmail.com wrote:
Hi Craig,
Thanks for the suggestion. I would if I could. I'd also probably try another file system. Though the good news is, enabling the write cache on that array has improved things significantly. Which, in my case, was:
tw_cli /c2/u0 set cache=on
Now, if only I had the battery backup unit for the card.
Thanks, everyone for their suggestions. For now I am happy with the situation, but I'd be interested to hear the experiences of others.
Austin
On Thu, Sep 1, 2011 at 3:58 PM, Craig White craig.white@ttiltd.com wrote:
On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
Hello,
Does anyone have experience using a 3ware 9650SE series raid controller
on CentOS 6.0?
use RAID 10
Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up. If performance is desired, RAID 5 is not the way to go.
Craig _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Thank you for the clarification Craig, I am re-examining my storage strategy, thus my email. Sadly this machine has been in service a number of years and already contains more data than is possible in a RAID 10 configuration. If I had the spare space and resources I'd be thrilled to switch to RAID 10. That is not the case, however. Given that, I am faced with the choice between having the machine entirely unusable between now and when I can acquire a BBU or using it with the risk of data loss/corruption, I will chose the latter. Neither data loss nor corruption would go unnoticed in the period that this will be necessary.
Does that sound entirely unreasonable?
Austin
On Fri, Sep 2, 2011 at 10:51 AM, Craig White craig.white@ttiltd.com wrote:
Data loss could conceivably occur on shutdown or restart too - just saying... You are assuming that the data that doesn't get written to disk is going to be non-essential... I wish you good luck with that. I think if one doesn't want to be an idiot, one would not enable a cache that has no means to ensure that the cache is written to disk.
I think your take away from all of this is somewhat misdirected. Not having a BBU simply means that your writes really should always be synchronous/immediate. That shouldn't really be a problem and shouldn't impose a large performance penalty.
Your performance issue relates more to the fact that RAID 5 implementation on the 3Ware cards is rather poor and modes such as RAID 10 (RAID 0 + 1) will give you much more speed that you realize. If you also consider on the surprisingly higher rates of failure with loss of data possibility when reconstructing a missing/dead drive on a RAID 5 setup you really should be re-examining your storage strategy.
Craig
On Sep 1, 2011, at 5:43 PM, Austin Godber wrote:
At this point the card is pretty much useless without that cache enabled.
Without recommendations for making writes of 256MB or larger files faster without this cache enabled, I will have to accept the possible data loss in the event of power outage. If it is only the case of data loss during a power outage, I will take that ... rather than failure to write at all during 99% of my usage.
I will, for the sake of not being an idiot, look into buying the BBUs.
Austin
On Thu, Sep 1, 2011 at 4:49 PM, Tom Bishop bishoptf@gmail.com wrote: Keep in mind you really only want to enable the cache if you have a bbc, otherwise you are risking your data since it can/will cache writes...just something to keep in mind.
On 9/1/11, Austin Godber godber@gmail.com wrote:
Hi Craig,
Thanks for the suggestion. I would if I could. I'd also probably try another file system. Though the good news is, enabling the write cache
on
that array has improved things significantly. Which, in my case, was:
tw_cli /c2/u0 set cache=on
Now, if only I had the battery backup unit for the card.
Thanks, everyone for their suggestions. For now I am happy with the situation, but I'd be interested to hear the experiences of others.
Austin
On Thu, Sep 1, 2011 at 3:58 PM, Craig White craig.white@ttiltd.com
wrote:
On Sep 1, 2011, at 1:41 PM, Austin Godber wrote:
Hello,
Does anyone have experience using a 3ware 9650SE series raid
controller
on CentOS 6.0?
use RAID 10
Unless something has changed, RAID 5 is notoriously slow on the 3Ware controllers. Whatever you do will only incrementally speed things up.
If
performance is desired, RAID 5 is not the way to go.
Craig _______________________________________________ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
-- Craig White ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ craig.white@ttiltd.com 1.800.869.6908 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ www.ttiassessments.com
Need help communicating between generations at work to achieve your desired success? Let us help!
CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
On Fri, 2011-09-02 at 12:03 -0700, Austin Godber wrote:
Thank you for the clarification Craig, I am re-examining my storage strategy, thus my email. Sadly this machine has been in service a number of years and already contains more data than is possible in a RAID 10 configuration. If I had the spare space and resources I'd be thrilled to switch to RAID 10. That is not the case, however. Given that, I am faced with the choice between having the machine entirely unusable between now and when I can acquire a BBU or using it with the risk of data loss/corruption, I will chose the latter. Neither data loss nor corruption would go unnoticed in the period that this will be necessary.
Does that sound entirely unreasonable?
---- reminds me of dialog in 'So I Married an Axe Murderer'...
Charlie: "For example, how many people have you brutally murdered?"
Harriet: "Brutal" is a very subjective word. What's brutal to one person might be reasonable to somebody else."
I think SATA hard drives are so cheap nowadays, that it's hard to justify choosing RAID 5 for more yield at the cost of performance and reliability.
Craig
I think SATA hard drives are so cheap nowadays, that it's hard to justify choosing RAID 5 for more yield at the cost of performance and reliability.
Another point to consider with slower sata discs that are also large is the prolonged rebuild time and degraded redundancy during this rebuild, especially with r5 vs r6.
Additionally I can confirm as I also have several lsi cards and all perform like hell in r5/6 even with bbu.
jlc
Additionally I can confirm as I also have several lsi cards and all perform like hell in r5/6 even with bbu.
Is that the "fast as hell" or "slow as hell" kind?
I ask because I have a couple of IBM M5015 (rebranded LSI 9260-8i) controllers that I run in RAID-10 and as I'm somewhat on a budget for disks (coming home with $1200 worth of 15k SAS drives will get me shot) I'd rather run a RAID-5/6 array if the performance degradation is minimal. And yes, mine do have the BBU. ;-)
Not that the performance of an 8x80GB SATA-300 array is much to write home about in the first place, but for my purposes it works fine.
On Fri, 2011-09-02 at 21:14 -0700, Drew wrote:
Additionally I can confirm as I also have several lsi cards and all perform like hell in r5/6 even with bbu.
Is that the "fast as hell" or "slow as hell" kind?
I ask because I have a couple of IBM M5015 (rebranded LSI 9260-8i) controllers that I run in RAID-10 and as I'm somewhat on a budget for disks (coming home with $1200 worth of 15k SAS drives will get me shot) I'd rather run a RAID-5/6 array if the performance degradation is minimal. And yes, mine do have the BBU. ;-)
Not that the performance of an 8x80GB SATA-300 array is much to write home about in the first place, but for my purposes it works fine.
---- I don't know how either of us could have made our opinions any more clear...
Don't use the 3ware/LSI SATA RAID controllers in RAID 5 or RAID 6 mode if performance and reliability are of concern. I only use hardware RAID because performance and reliability are my concern. I am told that the Areca cards are much better on RAID 5/6 but I have no first hand experience with them. While you may get better performance with write-back caching (don't enable without a BBU), the improvement is incremental. Stick with RAID 10.
Craig
I don't know how either of us could have made our opinions any more clear...
Just curious dude. "perform like hell" could go either way.
Don't use the 3ware/LSI SATA RAID controllers in RAID 5 or RAID 6 mode if performance and reliability are of concern. I only use hardware RAID because performance and reliability are my concern. I am told that the Areca cards are much better on RAID 5/6 but I have no first hand experience with them. While you may get better performance with write-back caching (don't enable without a BBU), the improvement is incremental. Stick with RAID 10.
Somebody at IBM must not have gotten the memo then as the M5015/9260-8i is their top of the line on-board RAID controller. :-)
Is that the "fast as hell" or "slow as hell" kind?
Slow.
I ask because I have a couple of IBM M5015 (rebranded LSI 9260-8i) controllers that I run in RAID-10 and as I'm somewhat on a budget for disks (coming home with $1200 worth of 15k SAS drives will get me shot) I'd rather run a RAID-5/6 array if the performance degradation is minimal. And yes, mine do have the BBU. ;-)
Not that the performance of an 8x80GB SATA-300 array is much to write home about in the first place, but for my purposes it works fine.
For home, I doubt you would have an issue.
-
Austin Godber -
Craig White -
Craig White -
Drew -
Joseph L. Casale -
Marcelo Beckmann -
Tom Bishop