[CentOS] mirroring with lftp
R P Herrold
herrold at owlriver.com
Mon Jun 27 22:58:58 UTC 2011
On Mon, 27 Jun 2011, John R Pierce wrote:
> does someone have a script for maintaining a repo with lftp they'd like
> to share? what I saw on the wiki wasn't very helpful.
[root at xps400 ~]# grep kernel *conf
lftp-centos-4-updates.conf:
ftp://mirrors.kernel.org/centos/4/updates/i386 \
lftp-centos-4-updates.conf:
ftp://mirrors.kernel.org/centos/4/updates/x86_64\
lftp-centos-5-updates.conf:
ftp://mirrors.kernel.org/centos/5/updates/i386 \
lftp-centos-5-updates.conf:
ftp://mirrors.kernel.org/centos/5/updates/x86_64\
lftp-openwall.conf:
ftp://mirrors.kernel.org/openwall/Owl/contrib/2.0/SRPMS/ \
[root at xps400 ~]# cat lftp-centos-5-updates.conf
#
# Get the Centos 5 updates
#
mirror -c -e \
ftp://mirrors.kernel.org/centos/5/updates/i386 \
/var/ftp/pub/mirror/centos/centos-5/5/updates/i386/RPMS
#
mirror -c -e \
ftp://mirrors.kernel.org/centos/5/updates/x86_64 \
/var/ftp/pub/mirror/centos/centos-5/5/updates/x86_64/RPMS
#
# The following line is for our yum-arch and
# createrepo logic
#
# yum: /var/ftp/pub/mirror/centos/centos-4/4/updates/i386/RPMS
#
[root at xps400 ~]#
This is warpped in a driver script that walks through the
directory, lokking for files ending in .conf -- when it finds
them it runs:
lftp -f /root/lftp-centos-5-updates.conf
which script is linked into /etc/cron.daily/
I do a lot more as well (lockfiles, deltas, emailled reporting
of unexpected variances, and so forth) so extract its essence
here
A google search with:
site:orcorc.blogspot.com lftp
will turn up relevant links
-I and -X globbing are arcane, and here are some examples I
use
[root at xps400 ~]# cat lftp-RHEL-enterprise-all-srpms.conf
#
#
#
mirror -c \
-I "*/SRPMS/*" \
ftp://ftp.redhat.com/pub/redhat/linux/enterprise/ \
/var/ftp/pub/mirror/redhat/rhel/at-release/SRPMSonly
#
mirror -c \
-I "*/SRPMS/*" \
ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/ \
/var/ftp/pub/mirror/redhat/rhel/updates/SRPMSonly
#
# new in 5
mirror -c \
-I "*/SRPMS/*" \
ftp://ftp.redhat.com/pub/redhat/linux/updates/rhn/ \
/var/ftp/pub/mirror/redhat/rhel/updates/rhn/SRPMSonly
#
mirror -c \
-I "*/SRPMS/*" \
ftp://ftp.redhat.com/pub/redhat/linux/eal/ \
/var/ftp/pub/mirror/redhat/rhel/eal/SRPMSonly
#
mirror -c \
-I "*/SRPMS/*" \
-X "*/4AS/*" \
-X "*/4Desktop/*" \
-X "*/4ES/*" \
-X "*/4WS/*" \
-X "*/5Client/*" \
-X "*/5Server/*" \
-X "*/RHHPC/*" \
-X "RHHPC/*" \
ftp://ftp.redhat.com/pub/redhat/linux/beta/ \
/var/ftp/pub/mirror/redhat/rhel/beta/SRPMSonly
#
# added X for RHHPC 2011-06-20
#
That last -X specification seemed to be needed (although one
out think the entry above it would provide the same effect).
It does not, as 'lftp' uses a textual, rather than a logical
parsing, and
"*/RHHPC/*" != "RHHPC/*"
as the match on the leading "*/" is not discarded
I wish it used a reasonable regex language, but as John
Boehner said last week in a different context:
'If ands, if's and but's were candies and nuts, it'd
be Christmas every day'
and I have no intention of forking lftp ;)
-- Russ herrold
More information about the CentOS
mailing list