On 10/12/2014 1:04 PM, Digimer wrote:
If it's only 1 Gbps, then the maximum sustainable write speed is ~120 MB/sec (at most; the slower of network or disk determines the max speed). You want the sync rate to be ~30% of maximum speed, or else you will choke out the apps using the DRBD resource and cause them to suffer a huge performance hit. App performance is (max write - sync rate) when a background sync is performed. This is why the default speed is very low.
In your case, you've set the sync rate way above what is possible, and I've seen this dramatically hurt performance and even full-on stall the sync operation. I would set this to '40M' at most.
verify apparently uses this same speed to limit how fast it runs... since its doing checksums on each side, its not actually transferring any data across the network, hence why I boosted it to 200M (that left the node that initiated the verify with one process at 90% CPU, whihc I figure is as fast as it can go)... as I said, I've set it back to 50M now that the verify completed.
btw, originally I had the verify-alg as sha1... my xeon X5650 2.7Ghz could only verify about 23MB/sec. setting it to crc32c sped up to 180-200MB/sec, which is more reasonable considering this is a 16TB volume built from 2 8x3TB drive raid6's striped together, behind a lsi 9261-8i megaraid sas2 card with 512MB BBU cache.