[CentOS] 40TB File System Recommendations

Thu Apr 14 15:26:57 UTC 2011
Benjamin Franz <jfranz at freerun.com>

On 04/14/2011 08:04 AM, Christopher Chan wrote:
>> Then try both for your use case and your hardware. We have wide raid6 setups
>> that does well over 500 MB/s write (that is: not all raid6 writes suck...).
> /me replaces all of Peter's cache with 64MB modules.
> Let's try again.

If you are trying to imply that RAID6 can't go fast when write size is 
larger than the cache, you are simply wrong. Even with just a 8 x RAID6, 
I've tested a system as sustained sequential (not burst) 156Mbytes/s out 
and 387 Mbytes/s in using 7200 rpm 1.5 TB drives. Bonnie++ results 
attached. Bonnie++ by default uses twice as much data as your available 
RAM to make sure you aren't just seeing cache. IOW: That machine only 
had 4GB of RAM and 256 MB of controller cache during the test but wrote 
and read 8 GB of data for the tests.

Version  1.96       ------Sequential Output------ --Sequential Input- 
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  
/sec %CP
xxxx            8G   248  99 155996  74 85600  42   961  99 386900  62 
628.3  29
Latency             33323us     224ms    1105ms   19047us   77599us     
Version  1.96       ------Sequential Create------ --------Random 
xxxx               -Create-- --Read--- -Delete-- -Create-- --Read--- 
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 17395  56 +++++ +++ 23951  61 27125  84 +++++ +++ 
32154  84
Latency               330us     993us     980us     344us      64us      

Benjamin Franz