[CentOS] rsync stuck at +- 50 MB/s, cp and scp are +- 200 MB/s

Tue Aug 11 17:07:17 UTC 2015
Juan Bernhard <juan at inti.gob.ar>

El 11/08/2015 a las 06:37 a.m., Götz Reinicke - IT Koordinator escribió:
> Hi,
>
> i have two servers, connected to to the lan by 10Gb with 10Gb and DAS
> hardware raid.
>
> Each system con read and write locally or to the 10G iscsi by more than
> 200 MB/s.
>
> Now I have to transfer backups form A to B and doing this with rsync
> always stuck at +- 48-50MB/s no matter which options, compressions,
> encryption etc I use. Even the plain default rsync is at that 50 Mb limit.
>
> coyp by scp goes up to 200 MB/s.

>
> Copy from and to my workstation with scp from or to both servers is at
> 1Gb limit (so +- 100 MB/s)
>
>
> Why is rsync stuck at +- 50 MB/s ? Any suggestions hints ...

rsync is not a like a copy, it checks from both sides that the file to 
copy is not the same on the other side. Copying something is straight 
forward, it dont wait to a remote check of the files metadata or 
attributes (last modification, permissions, etc). Access disk time could 
explain this...
I dont know how you are using rsync... if you are using the network 
protocol or a nfs mount.
check this also, if the rsync is using a single tcp connection for each 
file, and you have a lot of small files, the problem could be the slow 
start congestion algorithm of tcp (scp uses a single conection to copy, 
im sure of that. i dont know how rsync protocol works)
Also, if you have a lot of small files, it will take a lot of access 
time (disk seek time) for each to fetch this information and then start 
the copy.

See if you can do a iostat on each server doing a scp and a rsync to see 
the disk usage diference... if you find that you have a large queue on 
disk when using rsync, thats the problem...


I hope this helps to point the way to solve your problem.
Saludos, Juan.