[CentOS] mirroring with lftp

On Mon, 27 Jun 2011, John R Pierce wrote:

> does someone have a script for maintaining a repo with lftp they'd like
> to share?   what I saw on the wiki wasn't very helpful.

[root at xps400 ~]# grep kernel *conf
lftp-centos-4-updates.conf:
 	ftp://mirrors.kernel.org/centos/4/updates/i386 \
lftp-centos-4-updates.conf:
 	ftp://mirrors.kernel.org/centos/4/updates/x86_64\
lftp-centos-5-updates.conf:
 	ftp://mirrors.kernel.org/centos/5/updates/i386 \
lftp-centos-5-updates.conf:
 	ftp://mirrors.kernel.org/centos/5/updates/x86_64\
lftp-openwall.conf:
 	ftp://mirrors.kernel.org/openwall/Owl/contrib/2.0/SRPMS/  \
[root at xps400 ~]# cat lftp-centos-5-updates.conf
#
#       Get the Centos 5 updates
#
mirror -c -e \
         ftp://mirrors.kernel.org/centos/5/updates/i386          \
         /var/ftp/pub/mirror/centos/centos-5/5/updates/i386/RPMS
#
mirror -c -e \
         ftp://mirrors.kernel.org/centos/5/updates/x86_64                \
         /var/ftp/pub/mirror/centos/centos-5/5/updates/x86_64/RPMS
#
#       The following line is for our yum-arch and
#       createrepo logic
#
# yum: /var/ftp/pub/mirror/centos/centos-4/4/updates/i386/RPMS
#
[root at xps400 ~]#

This is warpped in a driver script that walks through the 
directory, lokking for files ending in .conf -- when it finds 
them it runs:
 	lftp -f /root/lftp-centos-5-updates.conf

which script is linked into /etc/cron.daily/

I do a lot more as well (lockfiles, deltas, emailled reporting 
of unexpected variances, and so forth) so extract its essence 
here

A google search with:
 	site:orcorc.blogspot.com lftp
will turn up relevant links

-I and -X globbing are arcane, and here are some examples I 
use

[root at xps400 ~]# cat lftp-RHEL-enterprise-all-srpms.conf
#
#
#
mirror -c  \
         -I "*/SRPMS/*"                                          \
         ftp://ftp.redhat.com/pub/redhat/linux/enterprise/       \
         /var/ftp/pub/mirror/redhat/rhel/at-release/SRPMSonly
#
mirror -c  \
         -I "*/SRPMS/*"                  \

ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/ \
         /var/ftp/pub/mirror/redhat/rhel/updates/SRPMSonly
#
#       new in 5
mirror -c  \
         -I "*/SRPMS/*"                  \
         ftp://ftp.redhat.com/pub/redhat/linux/updates/rhn/ \
         /var/ftp/pub/mirror/redhat/rhel/updates/rhn/SRPMSonly
#
mirror -c  \
         -I "*/SRPMS/*"                  \
         ftp://ftp.redhat.com/pub/redhat/linux/eal/ \
         /var/ftp/pub/mirror/redhat/rhel/eal/SRPMSonly
#
mirror -c  \
         -I "*/SRPMS/*"                                  \
         -X "*/4AS/*"                                    \
         -X "*/4Desktop/*"                               \
         -X "*/4ES/*"                                    \
         -X "*/4WS/*"                                    \
         -X "*/5Client/*"                                \
         -X "*/5Server/*"                                \
         -X "*/RHHPC/*"                                  \
         -X "RHHPC/*"                                    \
         ftp://ftp.redhat.com/pub/redhat/linux/beta/ \
         /var/ftp/pub/mirror/redhat/rhel/beta/SRPMSonly
#
#       added X for RHHPC 2011-06-20
#

That last -X specification seemed to be needed (although one 
out think the entry above it would provide the same effect). 
It does not, as 'lftp' uses a textual, rather than a logical 
parsing, and
 	"*/RHHPC/*" != "RHHPC/*"
as the match on the leading "*/" is not discarded

I wish it used a reasonable regex language, but as John 
Boehner said last week in a different context:
 	'If ands, if's and but's were candies and nuts, it'd
 	be Christmas every day'

and I have no intention of forking lftp  ;)

-- Russ herrold