[CentOS-mirror] Mirror not listed for updates x86_64 version 6

Tue Jun 30 09:40:20 UTC 2015
Fabian Arrotin <arrfab at centos.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 30/06/15 09:57, Grzegorz Paszka wrote:
> W dniu 2015-06-18 16:43, Fabian Arrotin pisze:
>> I can have a look, but don't forget that the mirror crawler
>> process will do the following : - - query for mirrors from a
>> current country (but in random way) - - validate each mirror from
>> that randomized list until it gets to 10 mirror for each list
>> (per release/repo/arch) - - when 10 nodes are there, stop
>> 
>> So each time a yum mirrorlist is produced, your mirror can 
>> enter/leave/enter it again (and each mirrorlist is produced in
>> loop , and takes ~30 minutes), and there are more than 10 mirrors
>> in .pl actually :-)
>> 
>> Hope that it answers your question
>> 
>> 
> I decide to take a deep look at which mirrors are presented by 
> http://mirrorlist.centos.org/..........
> 
> I logged every 5 minutes response from mirrorlist website for
> client from Poland for such repos:
<snip>

> 
> First conclusion. mirrorlist randomization algorithm is not working
> very well or something else is going bad. Some mirrors are
> definitely more popular that other. I.e centos,slaskdatacenter.com
> vs centos.komster.pl or vs ftp.pbone.net (which is synced 12 times
> a day and has 100% uptime and availability during test)

Well, the first thing to understand is that there is *no* mirrorlist
randomization algorithm : the crawler script just do a mysql query,
and with "ORDER BY RAND()" , nothing more

> 
> Second conclusion. During test I noticed that mirrorlist validation
> has some problems because for all update repos and x86_64 extras 7
> repo I have in log file that for some time mirrorlist returns
> couple mirror sites outside Poland. It didn't happened for other
> repos. Based on that I assume that every mirror was up (excluding
> centos.po.opole.pl) and even that your mirrorlist page returned
> mirrors outside Poland. If it'll help to investigate problem I can
> provide date and time of such situations.

As the the mirrorlist crawler script run in "loop" and for reach
release in parallel, those are changing quite fast.
As there were quite some updates recently, it makes sense that [os]
remains intact while [extras] and [updates] repodata (repomd.xml)
change often.

> 
> 
> I think that limit of 10 mirrors on mirrorlist site is not
> profitable for users. Very good example is USA. Centos has 145
> mirrors in USA. If you present only 10 it's probably that users use
> not optimal mirror sites for them. Yum has fastestmirror plugin,
> let the user to choose the best mirror for him.

True, and the 10 mirror limit is quite old (from where the scripts
were written, and don't known the real reason) but it does that for
every country, and not on a "country basis".
Logic is the following : it tries for a country to find at least 10
nodes that are "current " (so providing the same repodata as on
master, or at least in cache), and if it can't , it add some nodes
from nearby countries (which are up2date)

We can discuss that limit at the moment, but don't forget that the
more you add mirrors in the the list, the more outgoing checks
yum-fastestmirror-plugin will also do .. so finding the correct
balance is always a difficult game.

Now, for people managing a bunch of nodes, and willing to use the
faster/nearest all the time, $cfgmgmt is the solution that all
sysadmins/companies are using to modify the yum config to point to a
specific server.
Or (what we also do within centos infra), you have a local dns
resolver that point mirrorlist.centos.org to a local server with a 5
lines of php code redirecting you to the server you want too, without
having to modify the yum config at the client side.

> 
> Best regards. Grzegorz
> 

Cheers,

PS : there are some points we'd like to address in the future for
mirrorlists, as for example the fact that ipv4 and ipv6 mirrorlists
nodes aren't even using the same code.
If you follow the centos-devel list, you know that we're implementing
(and opening up soon) FAS as a central auth system for centos.org.
Then we can evaluate (again) MirrorManager, as its main requirement
was a central auth, like FAS.

- -- 

Fabian Arrotin
The CentOS Project | http://www.centos.org
gpg key: 56BEC54E | twitter: @arrfab
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlWSZAQACgkQnVkHo1a+xU7dVACgh82M9+xAMUTOWbMaNnrmkibs
AWAAnRM1/ql7GmbHnTMp0o/JFUeGQEPM
=5MBU
-----END PGP SIGNATURE-----