[CentOS-mirror] IRC meeting regarding new mirroring system for CentOS

Sun Nov 14 18:14:50 UTC 2010
Peter Pöml <peter at poeml.de>

Hi,

MirrorBrain with Yum mirrorlist support is released now.
http://mirrorbrain.org/news/2150-support-yum-mirror-lists/

Furthermore, I have set up a test instance on my little host. You find
it here: http://centos.mirrorbrain.org/

You can play with it and see if it behaves as expected.

It takes the usual Yum queries, although I should note that I added only
some mappings (those that I knew of:

MirrorBrainYumDir release=(4|4\.8|5|5\.5) repo=(os|extras|addons|updates|centosplus|contrib) \
			arch=i586 $1/$2/i386    repodata/repomd.xml
MirrorBrainYumDir release=(4|4\.8|5|5\.5) repo=(os|extras|addons|updates|centosplus|contrib) \
			arch=x86_64 $1/$2/x86_64  repodata/repomd.xml


I imported all the mirrors (which I found on the centos website), minus
three or four whose entries in the html table were not quite correct.
That's 334 mirrors, of which ~323 seem to be reachable. I didn't do a
lot of checking obviously. So if your mirror isn't there, or with wrong
location, don't panic :-)

Many mirrors don't have any URLs listed than HTTP, so scanning them is
rather inefficient, and may not work in some cases. (I implemented
scanning of nginx indexes today because there were so many of them.
However, it is much more efficient to scan over rsync or FTP.)
With a mirror network that large, it is really very helpful when rsync
or FTP are available (best is rsync), because resources become a
limiting factor with such a large number of mirrors.

Some mirrors were listed with round-robin DNS names, which obviously
means that there is no guarantuee that the scanned mirror is the one
that a client is redirected to. It is better to handle these mirrors as
separate machines (unless it can be guarantueed that they are tightly
synced), especially if they are in separate regions (as sometimes is the
case). So, the special handling required here has not been considered in
my setup yet.

I fixed a number of broken (mostly rsync) URLs.

My setup contains no file hashes at all (the hashes that you might have
seen in other MirrorBrain instances). That's because I don't have enough
space on my disk for the CentOS file tree -- I have just a pseudo file
tree consisting of sparse files filled with zeroes. Thus, no useful
hashes can be served of course. But the rest is fully functional.

So, now your feedback would obviously be very interesting!

(Let me know if I should create further query->directory mappings to
enable more real-world testing!)

Peter
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: not available
URL: <http://lists.centos.org/pipermail/centos-mirror/attachments/20101114/1708df28/attachment-0004.sig>