On 14/10/14, 6:45, Peter Kjellström wrote: > On Mon, 13 Oct 2014 20:15:11 -0500 > Joakim Ziegler <joakim at terminalmx.com> wrote: > ... >> So, if I use iozone -a to test write speeds on the raw device, I get >> results in the 500-800MB/sec range, depending on write sizes, which >> is about what I'd expect. >> However, when I have an ext4 filesystem on this device, mounted with >> noatime and data=writeback, (the filesystem is completely empty) and >> I test with dd, the results are less encouraging: > ... >> Now, I'm sure there can be many reasons for this, but I wonder where >> I should start looking to debug this. > First I'd suggest comparing apples to apples. That is try doing the dd > test on the raw device and compare to dd on ext4. > Then you may want to try changing io scheduler from the default cfq to > deadline. This typically works better for many raid controllers but > ymmv. > Also testing with xfs instead of ext4 is probably worth it. xfs usually > outperform ext4 in streaming writes (like dd). Of course this raises > the question of whether that dd is a useful metric for your actual > load... xfs may infact be needed (3T * 7 = 21 TB > ext4 max (if I > remember correctly, refer to rh online data for rhel6 to make sure)). Upgrading to 6.5 with its new kernel did not fix the problem. I will be doing some more testing. The strange thing is, I have a near-identical machine also running CentOS 6.5, also with ext4 on the same controller (and another, newer Areca controller), and there it's extremely fast, on the fastest controller there, dd hits around 2GB/sec sustained over 200 GB of data on a 24-disk RAID6 (both systems have 96GB of RAM each). And yes, I've formatted with a newer version of e2fsprogs than is included with the distro, to get 16TB+ support, although in the case of the device I'm currently testing, it actually has two partitions, so I wouldn't have needed to. I'll do a bit more testing and come back with my results. -- Joakim Ziegler - Supervisor de postproducción - Terminal joakim at terminalmx.com - 044 55 2971 8514 - 5264 0864