[CentOS] rsync question: building list taking forever

Mon Oct 20 15:28:42 UTC 2014
Les Mikesell <lesmikesell at gmail.com>

On Sun, Oct 19, 2014 at 9:05 PM, Tim Dunphy <bluethundr at gmail.com> wrote:
>>
>> > Don't forget that the time taken to build the file list is a function of
>> > the number of files present, and not their size. If you have many
>> millions
>> > of small files, it will indeed take a very long time. Over sshfs with
>> > a slowish link, it could be days.
>> >
>> > ....and it may end up failing silently or noisily anyway.
>
>
> Ahhh, but isn't that part of the beauty of adventure that being a linux
> admin is all about? *twitch*

There's not that much magic involved.  The time it takes rsync to read
a directory tree to get started should approximate something like
'find /path -links +0"   (i.e. something that has to read the
directory tree and the associated inodes).    Pre-3.0 rsync versions
read and transfer the whole thing before starting the comparison and
might trigger swapping if you are low on RAM.

So, you probably want the 'source' side of the transfer to be local
for faster startup. But... in what universe is NFS mounting across
data centers considered more secure than ssh?  Or even a reasonable
thing to do?  How about a VPN between the two hosts?

-- 
   Les Mikesell
     lesmikesell at gmail.com