[CentOS] 40TB File System Recommendations

[Benjamin Franz]

On 04/14/2011 09:00 PM, Christopher Chan wrote:
>
> Wanna try that again with 64MB of cache only and tell us whether there
> is a difference in performance?
>
> There is a reason why 3ware 85xx cards were complete rubbish when used
> for raid5 and which led to the 95xx/96xx series.
> _

I don't happen to have any systems I can test with the 1.5TB drives 
without controller cache right now, but I have a system with some old 
500GB drives  (which are about half as fast as the 1.5TB drives in 
individual sustained I/O throughput) attached directly to onboard SATA 
ports in a 8 x RAID6 with *no* controller cache at all. The machine has 
16GB of RAM and bonnie++ therefore used 32GB of data for the test.

Version  1.96       ------Sequential Output------ --Sequential Input- 
--Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
pbox3        32160M   389  98 76709  22 91071  26  2209  95 264892  26 
590.5  11
Latency             24190us    1244ms    1580ms   60411us   69901us   
42586us
Version  1.96       ------Sequential Create------ --------Random 
Create--------
pbox3               -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 10910  31 +++++ +++ +++++ +++ 29293  80 +++++ +++ 
+++++ +++
Latency               775us     610us     979us     740us     370us     
380us

Given that the underlaying drives are effectively something like half as 
fast as the drives in the other test, the results are quite comparable.

Cache doesn't make a lot of difference when you quickly write a lot more 
data than the cache can hold. The limiting factor becomes the slowest 
component - usually the drives themselves. Cache isn't magic performance 
pixie dust. It helps in certain use cases and is nearly irrelevant in 
others.

-- 
Benjamin Franz