[CentOS] Best practices for copying lots of files machine-to-machine

Thu May 18 13:53:55 UTC 2017
m.roth at 5-cent.us <m.roth at 5-cent.us>

Vanhorn, Mike wrote:
>
> On 5/17/17, 5:27 PM, "CentOS on behalf of m.roth at 5-cent.us"
> <centos-bounces at centos.org on behalf of m.roth at 5-cent.us> wrote:
>
>>Why? I just rsync'd 159G in less than one workday from one server to
>>another. Admittedly, we allegedly have a 1G network, but....
>
> Well, I’ve don’t recall ever having to rsync more than 100G (although I am
> doing multiple rsyncs of about 86G as we speak), and I’ve never been able
> to do it with machines on their own, isolated switch (so my rsync’s are
> competing with everything else on the network), and it’s been a while
> since I’ve actually tried it multiple ways and measured it, but in my
> experience I’ve never see the network outperform the system bus.

I wasn't saying the network outperformed the system bus. Most of the time,
though, I don't have that as a possibility. Usually, all the drive bays
are full and in use.

When we get to terabytes, that's at least overnight; but a few hundred gig
I can do in a day, if I start early.

    mark