OK thanks for your help with that :-)

On 8/3/62 14:56, Alpix wrote:

Hey Patrick,

thats my sync script.

    #!/bin/bash


    DESTPATH="/var/www/mirror/centos/"

    RSYNC=/usr/bin/rsync

    LOCKFILE=/tmp/rsync-centos.lock




    synchronize() {

        $RSYNC -rltvHa --delete-after --delay-updates --safe-links
    eu-msync.centos.org::CentOS "$DESTPATH"

    }




    if [ ! -e "$LOCKFILE" ]

    then

        echo $$ >"$LOCKFILE"

        synchronize

    else

        PID=$(cat "$LOCKFILE")

        if kill -0 "$PID" >&/dev/null

        then

            echo "Rsync - Synchronization still running"

            exit 0

        else

            echo $$ >"$LOCKFILE"

            echo "Warning: previous synchronization appears not to
    have finished correctly"

            synchronize

        fi

    fi


    rm -f "$LOCKFILE"

Greetings Alpix

Am Fr., 8. März 2019 um 08:47 Uhr schrieb Alpix <contact@alpix.eu mailto:contact@alpix.eu>:

Hello Patrick,

i have one.
I will send it  in a few minutes.

Greetings


Patrick Shaw <patrick@shaw.co.th <mailto:patrick@shaw.co.th>>
schrieb am Fr., 8. März 2019, 08:42:

    Hi,

    Does anyone have some examples of lock scripts/files for
    rsync? Due to
    slow international performance in this country I'm regularly
    seeing
    overlapping rsyncs, I need to get that locked down ASAP.

    Patrick


    _______________________________________________
    CentOS-mirror mailing list
    CentOS-mirror@centos.org <mailto:CentOS-mirror@centos.org>
    https://lists.centos.org/mailman/listinfo/centos-mirror

---

CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

You can use this tool. https://github.com/tuna/tunasync https://github.com/tuna/tunasync-scripts

2019년 3월 9일 (토) 오전 12:00, Dattatec Mirrors mirrors@dattatec.com님이 작성:

El Viernes 08/03/2019 a las 04:42, Patrick Shaw escribió:

...

Hi,

Does anyone have some examples of lock scripts/files for rsync? Due to slow international performance in this country I'm regularly seeing overlapping rsyncs, I need to get that locked down ASAP.

Patrick

Hi, I'd advise against using a lock file as f your scripts dies for whatever reason, the lock file might linger around and prevent future rsyncs from running.

Let me qoute a mail from David Richardson to Fedora's mirror mailing list with an alternative involving flock (last part) and some more general tips that might be useful for mirror admins:

"I had the same problem you did with rsync taking forever (and find, and ls, and httpd).

Changing the sysctl vm.vfs_cache_pressure made a night-and-day difference (default is 100, I set it to 10).

vm.vfs_cache_pressure controls caching of inode data versus file contents.

The default (and centerpoint) is 100. Values less than 100 favor inode data, values greater than 100 favors file contents. Do NOT set it to zero. My understanding that if you set it to zero, bad things will happen and you will eventually OOM.

With this change, all my metadata stays in cache. I have two million inodes in use, and this setting costs me about 4GB of RAM. A no-change Fedora rsync takes 20 seconds for 425GB of content in 950k files (I exclude development and SRPMS).

I use --delete to handle the .~tmp~ directories. If one of my runs aborts, the next run will clean up after it.

My script is basically rsync wrapped with flock (rather than trying to cobble together a lock-file system).

The 200 in the flock command (and again at the end) is just a filehandle number; it doesn't really matter what it is, as long as nothing else uses it. The file name at the end also doesn't much matter. The file needs to be writeable (or creatable if it doesn't exist), but nothing is written to it. There's also no need to remove it afterwards.

### SCRIPT BEGINS ### ( flock -n 200 || { echo "Script is already running. Aborting." ; exit 1 ; } # ... commands executed under lock ...

/usr/bin/rsync --progress -aHv --update --delete --delete-excluded \ --delete-after --delay-updates rsync://your/source /your/dest/path

/usr/bin/report_mirror

) 200>/tmp/lock.update-fedora ### SCRIPT ENDS ###

Hope you find this useful!"

BR,

Ricardo J. Barberis Senior SysAdmin / IT Architect DonWeb La Actitud Es Todo www.DonWeb.com _____ _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org https://lists.centos.org/mailman/listinfo/centos-mirror

El Viernes 08/03/2019 a las 16:09, David Richardson escribió:

I'm quoted! I'm famous! :)

With reason, your flock advise was thoroughly implemented in our infrastructure :)

And lately I (almost accidentally) realised the good effects of lowering vm.vfs_cache_pressure, reinforced after re-reading your email :)

There's two parts to what was quoted below:

1. Use flock to make sure you only have one update running. It'll do the right thing if something happens and the job exits abnormally. Someone else has already written a tool, so use their work instead of reinventing it yourself.

2. Tune vm.vfs_cache_pressure. In my experience, this makes the operations where you care about interactive performance (ls, du, find, rsync updates from upstream) MUCH faster, and has little impact on bulk-transfer performance.

Thanks, DR