[CentOS] network copy performance is poor (rsync) - debugging suggestions?

Wed Jan 28 23:39:50 UTC 2015
Gordon Messmer <gordon.messmer at gmail.com>

On 01/23/2015 01:44 AM, Götz Reinicke - IT Koordinator wrote:
> I do have two centos 6.6 servers. With a "performance optimized" rsync I
> get an speed of 15 - 20 MB/s

That *is* pretty slow for sustained writes.  Does the same rate hold 
true for individual large files as it does for lots of small ones?  What 
filesystem are you using on each side?

> rsync -aHAXxv --numeric-ids --progress -e "ssh -T -c arcfour -o
> Compression=no -x"

It's worth noting that -X and -A are going to perform filesystem IO that 
you don't see on SMB, because it isn't going to preserve/set ACLs and 
extended attributes (IIRC).  So, one possibility is that you're seeing a 
difference in rate because you're doing lots of small files and 
filesystem operations are relatively slow.

You might drop those two options and see how that affects the rate.  If 
you determine that those are the cause of the performance difference, 
you can turn them back on, understanding that there's a cost associated 
with preserving that data.

> Both servers have plenty of memory and cpu usage looks low.

Define low.  If you're using top and press '1' to expand the CPU lines, 
you'll probably see one cpu with higher "us" percentage, which is SSH 
encrypting the data.  What percentage is that?  Is there a large value 
in "sy" or "hi" on any CPU?  Probably not since you see good rates using 
'dd' and smb copies, but I've seen systems where interrupt processing 
was a major bottleneck, so I make it a standard check.