We have upgraded the rsync used on the centos mirrors to a version >= 2.6.4 so that we can use the --delay-updates flag for rsyncing.
This flag (if you have rsync >= 2.6.4) will download all the updates, then move them into place ... so for items like yum repos, there is a much shorter time that the repo is not usable during long updates. If you have rsync >= 2.6.4, we recommend you use this switch for CentOS rsyncs.
See the below e-mail for info:
While a CVS commit is in progress, for instance, other users still see the previous state; this is not true for a YUM repository.
Hmm. This sounds like the crux of the problem. If the mirroring software could be tricked into copying the repodata last, wouldn't this problem (and this thread) go away?
rsync does not allow you to specify an order, however rsync has 2 options. --delay-updates will update the mirror at the end of the sync, which is near atomic (this is functionality that Jeff Pitman wrote when I needed it for my repository) and you have an atomic-script that comes with rsync that hardlinks the tree, makes updates in that new tree and finally atomically puts it all back.
This one thing right there will help tremendously. Thanks for the pointer to --delay-updates (any idea which version of rsync this first appeared?).
OK guys ... the answer is that is was added in version 2.6.4 of rsync ... and that is newer than both CentOS-3 and CentOS-4 :(
BUT - this is such a good feature, we have upgraded the CentOS mirrors to have version 2.6.6 of rsync from Dag's repo:
http://dag.wieers.com/packages/rsync/
We recommend that if you are rsyncing from centos.org that you get the .el3 or .el4 version of rsync (depending on the version of your mirror) from above and then add the --delay-updates switch to your rsync script when rsyncing from us.
Many thanks to Dag Wieers for the info on this issue, for submitting the request to get this included in rsync ... and for his outstanding repos.
On Thursday 15 September 2005 05:22, Johnny Hughes wrote:
We have upgraded the rsync used on the centos mirrors to a version >= 2.6.4 so that we can use the --delay-updates flag for rsyncing.
This flag (if you have rsync >= 2.6.4) will download all the updates, then move them into place ... so for items like yum repos, there is a much shorter time that the repo is not usable during long updates. If you have rsync >= 2.6.4, we recommend you use this switch for CentOS rsyncs.
Yes, indeed a very useful note. However if some mirrors do not use a recent rsync version and cannot upgrade for some reason there is another way to "simulate" the --delay-updates thingie: using a 2-pass mirror script. This method is explained for the Debian AMD64 mirror: http://amd64.debian.net/~joerg/mirror.html.
Of course the first pass needs to be adjusted a little so the repodata.xml & friends will be excluded.
Mihai