[CentOS] rsync question: building list taking forever
Tim Dunphy
bluethundr at gmail.com
Mon Oct 20 02:05:42 UTC 2014
>
> > Don't forget that the time taken to build the file list is a function of
> > the number of files present, and not their size. If you have many
> millions
> > of small files, it will indeed take a very long time. Over sshfs with
> > a slowish link, it could be days.
> >
> > ....and it may end up failing silently or noisily anyway.
Ahhh, but isn't that part of the beauty of adventure that being a linux
admin is all about? *twitch*
On Sun, Oct 19, 2014 at 8:27 PM, Cliff Pratt <enkiduonthenet at gmail.com>
wrote:
> On Mon, Oct 20, 2014 at 7:57 AM, Steve Thompson <smt at vgersoft.com> wrote:
>
> > On Sun, 19 Oct 2014, Keith Keller wrote:
> >
> > I suspect that sshfs's relatively poor performance is having an impact
> >> on your transfer. I have a 30TB filesystem which I rsync over an
> >> OpenVPN link, and building the file list doesn't take that long (maybe
> >> an hour?). (The links themselves are reasonably fast; if yours are not
> >> that would have a negative impact too.)
> >>
> >
> > Don't forget that the time taken to build the file list is a function of
> > the number of files present, and not their size. If you have many
> millions
> > of small files, it will indeed take a very long time. Over sshfs with
> > a slowish link, it could be days.
> >
> > ....and it may end up failing silently or noisily anyway.
>
> Cheers,
>
> Cliff
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
--
GPG me!!
gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
More information about the CentOS
mailing list