[CentOS] wget and resulting filenames with mirror option
R P Herrold
herrold at owlriver.com
Wed Jul 28 21:36:38 UTC 2010
On Wed, 28 Jul 2010, Joseph L. Casale wrote:
> Anyone know a way around this? I am using the mirror option so I don't
> have to keep track of what to get making it simple.
my apology in advance for suggesting another tool, but, there
you are ...
use lftp for mirroring? wget can do mirroring but it is
clunky compared to lftp, and much harder to script. man pages
are worse, to my taste
Looking, it seems I have a bit over 1400 files driving lftp
that basically look like this on a local interior mirror
[root at xps400 ~]# wc -l *.conf *.conf_ | tail -1
1490 total
[root at xps400 ~]# cat lftp-ORC.conf
#
# inside at ftp.first, this is NOT automatically run
#
mirror -c -e \
ftp://ftp.owlriver.com/pub/local/ORC \
/var/ftp/pub/mirror/ORC
#
-------
One can invoke it as simply as:
lftp -f lftp-ORC.conf
and it will of course run non-root if one desires; it can be
wrappered up in a scriptlet (I use a flock scheme in the
wrappering so it does not overrun into a prior day's run;
mirroring runs continuously) ...
http://bugs.centos.org/view.php?id=4443
suggests a simple way to gather detail:
export YMD=`date +%Y%m%d`
cd $TOP
date > DIRECTORY_SIZES.txt
date > MANIFEST-${YMD}.txt
find . -type d -exec du -sh {} \; >> DIRECTORY_SIZES.txt
find . -type f \; >> MANIFEST-${YMD}.txt
and diff to taste, optionally automailing when more than the
top 'date' line changes
-- Russ herrold
More information about the CentOS
mailing list