[CentOS-devel] rsync speed (was Re: CentOS 6 status update)

[Charles Polisher]

Karanbir Singh wrote:
> Hi,
> 
> Andrzej Szymański wrote:
> > I don't know whether you use (and can use) the following hack to speed
> > up rsyncing the iso files:
> >
> > - you need to have the full (or mostly complete) package tree on the target
> 
> We do some sanity tests that span the buildsys, the intermediary host 
> and the final rsync targets. So its important that the first set of 
> contents is sync'd out from the buildsys as-is. Perhaps extra caution, 
> but we have had problems in the past ( even with rsync saying two files 
> are identical when they were clearly not ).
> 
> On the other hand, the ISOS can be fairly easily re-created from the 
> tree, so if the rpms tree are is already in place that would be easier 
> than going down the tar route.

I found out the hard way in a large production setting
about rsync parameters that were "new to me":

   --bwlimit=0  
                 (Don't throttle transfers that run "too quickly")

   --whole-file
                 (don't use the delta-xfer algorithm,
                 the whole file needs replacing)

   --sockopts=TCP_NODELAY
                (_might_ be useful, you'd have to test
                 if it helps on your system)

and apropos of well-defined behavior, you might have
overlooked:

   --delay-updates

                This option puts the temporary file from each
                updated file  into a holding directory until the
                end of the transfer, at which time all the files
                are renamed into place in rapid succession.   This
                attempts to make the updating of the files a
                little more atomic.
-- 
Charles Polisher