[CentOS] rsync question: building list taking forever

Mon Oct 20 00:27:41 UTC 2014
Cliff Pratt <enkiduonthenet at gmail.com>

On Mon, Oct 20, 2014 at 7:57 AM, Steve Thompson <smt at vgersoft.com> wrote:

> On Sun, 19 Oct 2014, Keith Keller wrote:
>
>  I suspect that sshfs's relatively poor performance is having an impact
>> on your transfer.  I have a 30TB filesystem which I rsync over an
>> OpenVPN link, and building the file list doesn't take that long (maybe
>> an hour?).  (The links themselves are reasonably fast; if yours are not
>> that would have a negative impact too.)
>>
>
> Don't forget that the time taken to build the file list is a function of
> the number of files present, and not their size. If you have many millions
> of small files, it will indeed take a very long time. Over sshfs with
> a slowish link, it could be days.
>
> ....and it may end up failing silently or noisily anyway.

Cheers,

Cliff