[CentOS] How to speed up Rsync transfers

> -----Original Message-----
> From: centos-bounces at centos.org 
> [mailto:centos-bounces at centos.org] On Behalf Of Dan Carl
> Sent: Sunday, February 24, 2008 18:54
> To: CentOS mailing list
> Subject: Re: [CentOS] How to speed up Rsync transfers
> 
> This is exactly the situation I'm trying to avoid.
> Right now its less than 2GB new/edited images a day so the 
> rsync backup 
> finishes before the script runs again.
> But I can't take it for granted that this will always be the case.
> Any ideas would be appreciated. What do you mean by local staging?

Well, we use lock files.

> I'd like the backup to run from 7pm to 7am and then if it 
> didn't finish to 
> resume again the next night.

We run it every 20 hours after compleation.

> That way when nothing was added/edited on the weekends the 
> backup can catch 
> up.



>From my atq:

#!/bin/sh
# atrun uid=0 gid=0
# mail     root 0
umask 22
interval=20\ hours; export interval
limit=--bwlimit=64; export limit
cd /var/spool/mirrors || {
         echo 'Execution directory inaccessible' >&2
         exit 1
}

./centos.sh

>From centos.sh

[root at host67 ~]# cd /var/spool/mirrors
[root at host67 mirrors]# cat centos.sh
lockfile=$0.lck

unset interactive
unset verbose
interval="20 hours"
verbose="-q"

while [ "$1" != "" ]
do case $1 in

        --bwlimit=*)
                limit="--bwlimit=64"
        ;;
        --interactive)
                interactive=true
        ;;
        -t)
                interval="$2"
                shift
        ;;
        --prune)
                export prune="--delete-excluded"
        ;;
        -v)
                verbose="-v -v --progress"
                interactive=true
        ;;
        *)
                echo "Usage: $0"
                exit 1
esac

shift

done


if ( set -o noclobber; echo "$$" > "$lockfile") 2> /dev/null;

then

   trap 'rm -f "$lockfile"; exit $?' INT TERM EXIT

#   echo \
   rsync -azH \
   --delete \
   rsync://mirrors.kernel.org/centos/ \
   --exclude-from=centos.excludes \
   $verbose \
   $limit \
   $prune \
   centos/


   rm -f "$lockfile"
   trap - INT TERM EXIT


   if [ "$interactive" != "true" ]
   then

        export limit
        export interval
        echo $0 | at now + 20 hours > /dev/null 2> /dev/null

   fi

fi

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information. If you
have received it in error, purge the message from your system and
notify the sender immediately.  Any other use of the email by you
is prohibited.