i've maintained a local centos repository at work using rsync, but it seems the corp honchos have decided to block rsync at our firewall, plus its never been 100% reliable, I'd get aborts on protocol errors sometimes several times before pulling down a complete new distro update.
i'm trying to figure out how to do this with lftp, and its got me somewhat stymied.
I'm testing with --dry-run like...
lftp -c 'open http://mirrors.kernel.org/centos/ && mirror -c -X */ia64 -X */s390 -X */alpha -X */SRPMS --dry-run'
but my -X excludes don't seem to work (I don't want/need itanium, s390, alpha, or SRPMs in my repo).
and, it doesn't seem to be aware of the symlinks from, for example, /5/ to /5.6/ and seems to want to pull down two copies instead (I understand that http doesn't make symlinks obvious). Should I use ftp:// instead of http:// or something?
does someone have a script for maintaining a repo with lftp they'd like to share? what I saw on the wiki wasn't very helpful.
On Jun 27, 2011, at 6:26 PM, John R Pierce pierce@hogranch.com wrote:
i've maintained a local centos repository at work using rsync, but it seems the corp honchos have decided to block rsync at our firewall, plus its never been 100% reliable, I'd get aborts on protocol errors sometimes several times before pulling down a complete new distro update.
i'm trying to figure out how to do this with lftp, and its got me somewhat stymied.
I'm testing with --dry-run like...
lftp -c 'open http://mirrors.kernel.org/centos/ && mirror -c -X*/ia64 -X */s390 -X */alpha -X */SRPMS --dry-run'
but my -X excludes don't seem to work (I don't want/need itanium, s390, alpha, or SRPMs in my repo).
and, it doesn't seem to be aware of the symlinks from, for example, /5/ to /5.6/ and seems to want to pull down two copies instead (I understand that http doesn't make symlinks obvious). Should I use ftp:// instead of http:// or something?
does someone have a script for maintaining a repo with lftp they'd like to share? what I saw on the wiki wasn't very helpful.
I have a wget repo mirroring script.
I'll dig it up and post it tonight when I get home.
-Ross
On 06/27/11 3:46 PM, Ross Walker wrote:
does someone have a script for maintaining a repo with lftp they'd like
to share? what I saw on the wiki wasn't very helpful.
hmm. this seems to be working.
cd /path/to/local/mirror lftp -c 'open ftp://mirrors.kernel.org/pub/ && mirror -c -x ia64 -x s390 -x s390x -x alpha -x SRPMS centos'
[using ftp instead of http, and using -x instead of -X (exclude instead of exclude-globs)]
On Mon, 27 Jun 2011, John R Pierce wrote:
does someone have a script for maintaining a repo with lftp they'd like to share? what I saw on the wiki wasn't very helpful.
[root@xps400 ~]# grep kernel *conf lftp-centos-4-updates.conf: ftp://mirrors.kernel.org/centos/4/updates/i386 \ lftp-centos-4-updates.conf: ftp://mirrors.kernel.org/centos/4/updates/x86_64\ lftp-centos-5-updates.conf: ftp://mirrors.kernel.org/centos/5/updates/i386 \ lftp-centos-5-updates.conf: ftp://mirrors.kernel.org/centos/5/updates/x86_64\ lftp-openwall.conf: ftp://mirrors.kernel.org/openwall/Owl/contrib/2.0/SRPMS/ \ [root@xps400 ~]# cat lftp-centos-5-updates.conf # # Get the Centos 5 updates # mirror -c -e \ ftp://mirrors.kernel.org/centos/5/updates/i386 \ /var/ftp/pub/mirror/centos/centos-5/5/updates/i386/RPMS # mirror -c -e \ ftp://mirrors.kernel.org/centos/5/updates/x86_64 \ /var/ftp/pub/mirror/centos/centos-5/5/updates/x86_64/RPMS # # The following line is for our yum-arch and # createrepo logic # # yum: /var/ftp/pub/mirror/centos/centos-4/4/updates/i386/RPMS # [root@xps400 ~]#
This is warpped in a driver script that walks through the directory, lokking for files ending in .conf -- when it finds them it runs: lftp -f /root/lftp-centos-5-updates.conf
which script is linked into /etc/cron.daily/
I do a lot more as well (lockfiles, deltas, emailled reporting of unexpected variances, and so forth) so extract its essence here
A google search with: site:orcorc.blogspot.com lftp will turn up relevant links
-I and -X globbing are arcane, and here are some examples I use
[root@xps400 ~]# cat lftp-RHEL-enterprise-all-srpms.conf # # # mirror -c \ -I "*/SRPMS/*" \ ftp://ftp.redhat.com/pub/redhat/linux/enterprise/ \ /var/ftp/pub/mirror/redhat/rhel/at-release/SRPMSonly # mirror -c \ -I "*/SRPMS/*" \
ftp://ftp.redhat.com/pub/redhat/linux/updates/enterprise/ \ /var/ftp/pub/mirror/redhat/rhel/updates/SRPMSonly # # new in 5 mirror -c \ -I "*/SRPMS/*" \ ftp://ftp.redhat.com/pub/redhat/linux/updates/rhn/ \ /var/ftp/pub/mirror/redhat/rhel/updates/rhn/SRPMSonly # mirror -c \ -I "*/SRPMS/*" \ ftp://ftp.redhat.com/pub/redhat/linux/eal/ \ /var/ftp/pub/mirror/redhat/rhel/eal/SRPMSonly # mirror -c \ -I "*/SRPMS/*" \ -X "*/4AS/*" \ -X "*/4Desktop/*" \ -X "*/4ES/*" \ -X "*/4WS/*" \ -X "*/5Client/*" \ -X "*/5Server/*" \ -X "*/RHHPC/*" \ -X "RHHPC/*" \ ftp://ftp.redhat.com/pub/redhat/linux/beta/ \ /var/ftp/pub/mirror/redhat/rhel/beta/SRPMSonly # # added X for RHHPC 2011-06-20 #
That last -X specification seemed to be needed (although one out think the entry above it would provide the same effect). It does not, as 'lftp' uses a textual, rather than a logical parsing, and "*/RHHPC/*" != "RHHPC/*" as the match on the leading "*/" is not discarded
I wish it used a reasonable regex language, but as John Boehner said last week in a different context: 'If ands, if's and but's were candies and nuts, it'd be Christmas every day'
and I have no intention of forking lftp ;)
-- Russ herrold
John R Pierce wrote:
does someone have a script for maintaining a repo with lftp they'd like to share? what I saw on the wiki wasn't very helpful.
I am using mrepo, and for number of repos use lftp, but from mrepo's config. It downloads packages and then creates repositories for the web server, and even allows creating one larger repo from several smaller. It also takes care of ISO mounting so you do not need base repo synced.
Ljubomir
-
John R Pierce -
Ljubomir Ljubojevic -
R P Herrold -
Ross Walker