[CentOS] 40TB File System Recommendations

Fri Apr 15 13:05:30 UTC 2011
Christopher Chan <christopher.chan at bradbury.edu.hk>

On Friday, April 15, 2011 07:24 PM, Benjamin Franz wrote:
> On 04/14/2011 09:00 PM, Christopher Chan wrote:
>> Wanna try that again with 64MB of cache only and tell us whether there
>> is a difference in performance?
>> There is a reason why 3ware 85xx cards were complete rubbish when used
>> for raid5 and which led to the 95xx/96xx series.
>> _
> I don't happen to have any systems I can test with the 1.5TB drives
> without controller cache right now, but I have a system with some old
> 500GB drives  (which are about half as fast as the 1.5TB drives in
> individual sustained I/O throughput) attached directly to onboard SATA
> ports in a 8 x RAID6 with *no* controller cache at all. The machine has
> 16GB of RAM and bonnie++ therefore used 32GB of data for the test.
> Version  1.96       ------Sequential Output------ --Sequential Input-
> --Random-
> Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP
> /sec %CP
> pbox3        32160M   389  98 76709  22 91071  26  2209  95 264892  26
> 590.5  11
> Latency             24190us    1244ms    1580ms   60411us   69901us
> 42586us
> Version  1.96       ------Sequential Create------ --------Random
> Create--------
> pbox3               -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
>                 files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
> /sec %CP
>                    16 10910  31 +++++ +++ +++++ +++ 29293  80 +++++ +++
> +++++ +++
> Latency               775us     610us     979us     740us     370us
> 380us
> Given that the underlaying drives are effectively something like half as
> fast as the drives in the other test, the results are quite comparable.

Woohoo, next we will be seeing md raid6 also giving comparable results 
if that is the case. I am not the only person on this list that thinks 
cache is king for raid5/6 on hardware raid boards and the using hardware 
raid + bbu cache for better performance one of the two reasons why we 
don't do md raid5/6.

> Cache doesn't make a lot of difference when you quickly write a lot more
> data than the cache can hold. The limiting factor becomes the slowest
> component - usually the drives themselves. Cache isn't magic performance
> pixie dust. It helps in certain use cases and is nearly irrelevant in
> others.

Yeah, you are right - but cache is primarily to buffer the writes for 
performance. Why else go through the expense of getting bbu cache? So 
what happens when you tweak bonnie a bit?