[CentOS] How to speed up Rsync transfers

Dan Carl danc at bluestarshows.com
Mon Feb 25 17:32:36 UTC 2008


> On Sun, 2008-02-24 at 17:53 -0600, Dan Carl wrote:
> > ----- Original Message ----- 
> > From: "William L. Maltby" <CentOS4Bill at triad.rr.com>
<snip list header and now irrelevant stuff>
 
> Don't be afraid to seek/request some kind of raid/NAS/SAN resource if
> the data is mission-critical, growing constantly and volatile. It may
> not be needed now, but look down the road so you don't get into a
> constant cycle of scrambling to keep up with needs.

Have a LVM/raid already. 
Just want to have an automated offsite backup to be on the safe side.
Have learned the hard way raid is not a subsitute for a backup.

<snip>

> In conjunction with "lock" files mentioned in another reply, you may be
> able to gain something by segmenting the local and remote rsync. This
> allows 1) concurrent *local* compression and rsync (if CPU/memory
> resources are sufficient to avoid unduly slowing the user's activities -
> again "man nice" to reduce the effects on users) and 2) easier
> management of the remote rsync start/stop on directory boundaries as the
> window is entered/exited. This may not be needed at all or may be of
> limited benefit.

Lock file sound like the way I'll go.
I'm going to stick with the hand-crafted stuff

> Lastly, see if it's possible to run the rsync during normal hours. If
> your site has upload of 750KB/sec and during 90% of the normal workday
> only a small percentage is consumed, take advantage by doing some of the
> rsync (maybe in small chunks) during these hours at low priority and
> throttled appropriately. Presuming that most of your activity is
> download, not upload during the normal workday, and knowing that most of
> the rsync activity will be upload, not download, there is an opportunity
> there.

Something I'll look at down the road.

> -- 
> Bill

Thanks Bill
Lots of good imformation
and thanks to all
Dan




More information about the CentOS mailing list