[CentOS-mirror] mirrorlist.centos.org Query

[Fabian Arrotin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/04/15 14:37, Justin Coffman wrote:
> I’m curious as to how queries to mirrorlist.centos.org are
> processed.
> 
> For example, querying mirrorlist from my CentOS mirror:
> 
> justin at bruce:~$ curl -s
> "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os" |
> sort ftp://ftp.ussg.iu.edu/linux/centos/7.1.1503/os/x86_64/ 
> http://centos.eecs.wsu.edu/7.1.1503/os/x86_64/ 
> http://centos.sonn.com/7.1.1503/os/x86_64/ 
> http://ftp.usf.edu/pub/centos/7.1.1503/os/x86_64/ 
> http://mirror.beyondhosting.net/CentOS/7.1.1503/os/x86_64/ 
> http://mirror.clarkson.edu/centos/7.1.1503/os/x86_64/ 
> http://mirror.metrocast.net/centos/7.1.1503/os/x86_64/ 
> http://mirror.san.fastserv.com/pub/linux/centos/7.1.1503/os/x86_64/
>
> 
http://mirror.trouble-free.net/centos/7.1.1503/os/x86_64/
> http://reflector.westga.edu/repos/CentOS/7.1.1503/os/x86_64/
> 
> Same query from my home system about a thousand miles away:
> 
> justin at selina:~$ curl -s
> "http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os" |
> sort ftp://ftp.ussg.iu.edu/linux/centos/7.1.1503/os/x86_64/ 
> http://centos.eecs.wsu.edu/7.1.1503/os/x86_64/ 
> http://centos.sonn.com/7.1.1503/os/x86_64/ 
> http://ftp.usf.edu/pub/centos/7.1.1503/os/x86_64/ 
> http://mirror.beyondhosting.net/CentOS/7.1.1503/os/x86_64/ 
> http://mirror.clarkson.edu/centos/7.1.1503/os/x86_64/ 
> http://mirror.metrocast.net/centos/7.1.1503/os/x86_64/ 
> http://mirror.san.fastserv.com/pub/linux/centos/7.1.1503/os/x86_64/
>
> 
http://mirror.trouble-free.net/centos/7.1.1503/os/x86_64/
> http://reflector.westga.edu/repos/CentOS/7.1.1503/os/x86_64/
> 
> How is the list of responses to mirrorlist queries determined?
> Obviously, it can’t return the full mirrorlist by country or
> something, that wouldn’t be efficient when yum-fastestmirror runs.
> 
> _______________________________________________ CentOS-mirror
> mailing list CentOS-mirror at centos.org 
> http://lists.centos.org/mailman/listinfo/centos-mirror
> 

So already explained, but here we go (I'd probably need to just write
a wiki page and point people at it ...)
- - mirrors are entered into mysql
- - mirmon checks that mirrors are reachable and verify time/timestamp
at the root
- - validated mirrors (basic check) then appear on
http://mirror-status.centos.org
- - specific crawler processes (one per release)
  - do a mysql query for active and current mirrors validated by
mirmon (but "order by rand()" !)
  - checks each ${arch}/${repo} and validate that mirror. If good add
it to the list
  - stops when it reaches 10 good mirrors for a country
- - mirrorlists are then pushed/used by the mirrorlists.centos.org nodes

So, one thing to understand is the 10 mirrors limit for mirrorlist,
and the "order by rand()" in the sql query. What that means is that
depending on which mirrors are returned to the crawler process, they
can so appear/disappear from mirrorlist every ~60 minutes (time needed
to generate all mirrorlists for all countries for a specific release)
Obviously, countries with less than 10 mirrors don't notify that
effect in the mirrorlists.

Hope that it answers the question

- -- 

Fabian Arrotin
The CentOS Project | http://www.centos.org
gpg key: 56BEC54E | twitter: @arrfab
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iEUEARECAAYFAlUmM9QACgkQnVkHo1a+xU4U2ACWPzne4Nr5Yw91yXfSsbF/iG9d
2QCfaqyeeRZegdaO3xsFmOQJ4GMBNFk=
=pxQX
-----END PGP SIGNATURE-----