Hello everyone. As some of you may have noticed, I have started helping
Fabian Arrotin with mirrorlist management tasks. For example, I can now
make changes to the list of CentOS mirrors myself. I will also continue
maintaining the IPv6 mirror checker (see http://miuku.net/ipv6reach/ for
stats), which runs separate from the main CentOS mirror checker.
In order for me to change the access control list which specifies the IP
addresses that can connect to msync.centos.org nodes, the ACL had to be
moved from a separate configuration file to the main mirror database.
This gives me easier access to the ACL. The ACL is exported from the
database to msync nodes periodically.
The reason why I'm writing this message is that it's possible that some
of the IP addresses got accidentally assigned to mirrors that have been
disabled, and this causes that IP address to be excluded from the ACL. I
don't expect this to be a particularly widespread problem, but it is a
possibility.
If you have problems syncing (you get a message like Unknown module
'CentOS' and/or your timestamp.txt is too old), please let us know and
we'll fix the problem ASAP. Please specify your mirror's http URL and IP
addresses (IPv4, and if you're lucky enough to have IPv6, that too) so
that we could locate your mirror's data from the database quickly.
Thanks for your cooperation. If you have any other concerns or
questions, feel free to send me an email or come to #centos-mirror on
Freenode for a chat.
On 01/03/17 14:44, Anssi Johansson wrote:
> We have 600+ active CentOS mirrors[1] and sometimes there are critical
> updates that would need to get published to the mirror network as
> quickly as possible. As of now, it takes around five hours to go from 0
> up-to-date mirrors to 80% up-to-date mirrors, with a longer tail for the
> remaining 20%.
>
> Current guidelines for setting up a mirror[2] specify that mirrors
> should sync 2-4 times per day. Instead of telling mirrors to sync hourly
> I thought we could come up with something smarter.
>
> One option that I have considered would be something similar to what the
> ClamAV guys use to signal end users that there is new content. They use
> a DNS TXT record for that purpose. For example, as of this writing "dig
> txt +short current.cvd.clamav.net" produces
> "0.99.2:57:23148:1488371340:1:63:45637:290", which shows the version
> numbers for ClamAV itself, main virus database, daily virus database,
> timestamp and other version numbers.
>
> We could have something similar, showing the timestamps when the content
> for CentOS, CentOS AltArch and CentOS Vault was last modified, like
> "1488372781:1487767981:1488113581". "Last modified" in the sense that
> new packages got added at that time. The idea is that mirrors could set
> up scripts to check that timestamp from DNS more frequently (such as
> hourly) without causing load issues to msync nodes by rsyncing hourly.
> The TTL for the TXT record could be relatively small, like 10 minutes.
>
> As you're all aware, DNS is a prime example of a very scalable system,
> and that's why I'm fond of this solution. Another option would be to
> publish the same data in a central location and served over http(s), if
> relaying the timestamp data via DNS is not desired for some reason.
>
> The basic principle would be "if timestamp in TXT record > my current
> timestamp (TIME file), synchronize the mirror". With more frequent
> syncs, mirror admins would need to take care that no two rsync runs
> would happen at the same time. Using lockfile in the scripts would help
> with this. I hope that many of the mirror admins already use lockfiles,
> but providing an example script might help for the newer mirror admins.
>
> The timestamps should be updated only after it has been verified that
> all (or at least the majority) of msync nodes actually have the content.
> It takes a while for the data to reach all the msync nodes from the master.
>
> On the other hand, this may cause some traffic peaks for the msync
> nodes. I don't know how well they would handle the peaks. One obvious
> way to alleviate the peaks would be to instruct mirror admins to pick a
> random minute when to check for new content. echo $[ $RANDOM % 60 ]
> works nicely for this. I don't have statistics, but I believe there
> might be mirrors that sync at "0 */6" ie. at the top of the hour.
>
>
> If the traffic peaks to msync nodes is deemed to be a problem, there
> might be ways to reduce the load to msync nodes. The following idea
> could be implemented separately from the above timestamp idea, if needed.
>
> There could be some sort of a "web service" which instructs mirrors
> where to sync from. The core idea in this is that the source might not
> always be a msync.centos.org server, but it could also be a nearby
> public mirror that offers rsync and has been verified to have the new
> content. If requested from Finland, that service could say "ok, you're
> from .fi, go sync from ftp.funet.fi as it has the new content already"
> or "uh oh, no nearby external mirrors have the new content, please rsync
> from eu-msync.centos.org". It could simply return a list of rsync
> servers in descending priority, with some msync.centos.org addresses at
> the bottom as fallback. Once the mirror has rsynced, the mirror could
> ping back and say "I have the content now, please check, and if OK, add
> me to the list of mirrors that have the new content".
>
> One concern is that the list of rsync sources would need to be
> protected, so that mirrors could not be tricked into syncing from a
> malicious source (think DNS poisoning). Ways to protect from this
> include DNSSEC, TLS and PGP signed data.
>
>
> Any thoughts about this?
>
>
> [1] http://mirror-status.centos.org/
> [2] https://wiki.centos.org/HowTos/CreatePublicMirrors
Hi Anssi,
Thanks a lot for all those ideas.
While I understand your idea behind a TXT record at the DNS level, I'd
say that I'm not a fan of that idea. Even with lower TTL, you'd be
surprized how many hits we had on servers that were migrated to new
ones, as it seems some ISPs aren't obeying the TTL and so were still
serving wrong (and expired) A/AAAA records from their cache.
DNS itself (currently for centos.org) isn't DNSSEC enabled too, so that
would mean other protection, etc
I'd prefer your alternative with "let's host this behind a https web
server" and also for the reason that it's easier to have TLS for
webserver, and that from the automation point-of-view, it's easier for
people allowed to build/sign/push (two people) to just update/drop a
file somewhere, than using DNS modification.
For dns, as the zone is actually under git/puppet control, that would
mean *not* using that, but rather having a delegated zone that would
allow nsupdate with a key that those people would share, etc ... So the
simple file served from https seems easier from my side.
I'd like to get opinions from Johnny/KB (people able to sign/push) as
they'd be directly concerned by that decision.
From an "external mirror admin" PoV, we should also use the
centos-mirror list to discuss this, to get their opinions ?
Also, we can divide your proposal into two parts :
- external mirrors can check a file they can compare against to sync
"faster" than through their cron jobs (discussed above)
- modifying completely the msync.centos.org network to have external
mirrors not syncing from us, but betwen them (not sure how people feel
about this)
PS : Anssi is now part of the mirrors managers team for CentOS , for
people not yet aware of that fact
--
Fabian Arrotin
The CentOS Project | http://www.centos.org
gpg key: 56BEC54E | twitter: @arrfab
Is there a a minimum sync interval for syncing from the msync nodes? I know it’s recommended mirrors sync 4-6 times a day, so every 4-6 hours.
I’m an advocate of getting updates to users as quickly as possible. I’m not opposed to syncing my mirrors on an hourly basis as long as it won’t cause a load problem for the server upstream of me.
Thoughts?
Our mirror has a IPv6 address since about a week. Can our IPv6 address
please also be listed in the msync ACLs:
$ host ftp-stud.hs-esslingen.de
ftp-stud.hs-esslingen.de is an alias for rhlx01.hs-esslingen.de.
rhlx01.hs-esslingen.de has address 129.143.116.10
rhlx01.hs-esslingen.de has IPv6 address 2001:7c0:700::10
Adrian
Hello,
It looks like the mirror I did the initial sync from http://centos.mirror.lstn.net/ <http://centos.mirror.lstn.net/> is starting to run behind.
I’ve updated the mirror to use the msync servers and it’s up to date.
Thanks!
>27.2.2017, 23.52, James DeVincentis kirjoitti:
>> Sorry for the update already this mirror now has native IPv6 addressing.
>>
>> This mirror has IPv6 2604:6600:38b:0:104:149:238:196.
>>
>>> On Feb 25, 2017, at 4:22 PM, James DeVincentis <james.d at hexhost.net <https://lists.centos.org/mailman/listinfo/centos-mirror>
>>> <mailto:james.d at hexhost.net <https://lists.centos.org/mailman/listinfo/centos-mirror>>> wrote:
>>>
>>> Hello all,
>>>
>>> I have set up a mirror for public usage, details follow. Initial
>>> synchronize has been completed.
>>>
>>> HTTP: http://mirror.lps.rocks/centos/ <http://mirror.lps.rocks/centos/> https://mirror.lps.rocks/centos/ <https://mirror.lps.rocks/centos/>
>>> FTP: ftp://mirror.lps.rocks/centos/ <ftp://mirror.lps.rocks/centos/> ftps://mirror.lps.rocks/centos/ <ftps://mirror.lps.rocks/centos/>
>>> RSYNC: rsync://mirror.lps.rocks/centos/ <rsync://mirror.lps.rocks/centos/>
>>>
>>> Sync Schedule: every 6 hours
>>> Bandwidth: 1Gbps
>>> Location: US, TX, Dallas
>>> Sponsor: LPS
>>> Sponsor URL: https://lps.rocks <https://lps.rocks/>
>>> IPv4 address to authorize: 104.149.238.196
>>> IPv6 address to authorize: 2001:470:1f0f:7d1:104:149:238:196
>>> E-Mail Contact: admin at lps.rocks <https://lists.centos.org/mailman/listinfo/centos-mirror>
>>> Mirroring AltArch: No
>>>
>>> Thanks!
>
> Hi, thank you for submitting a new mirror. I have now added your mirror
> to the database, and 104.149.238.196 and 2604:6600:38b:0:104:149:238:196
> should now (or in the next hour) be in the ACL so you can sync directly
> from rsync://us-msync.centos.org/CentOS/ <rsync://us-msync.centos.org/CentOS/>
> I hope I got the right IPv6 address. I understood from your messages
> that 2604:6600:38b:0:104:149:238:196 is the correct IPv6 address you are
> syncing from, and the other IPv6 address is no longer in use. If this is
> not the case, please let me know.
> It may take a while for the ACL changes to propagate to all msync nodes,
> and for the mirror crawler to verify your mirror. Note that as of now,
> your mirror is slightly old, and as such it won't be included on the
> mirrorlist. The next time you sync and the mirror crawlers visit your
> mirror, your mirror will be listed in all appropriate places. The first
> appearance will be to http://mirror-status.centos.org/#us <http://mirror-status.centos.org/#us>
> Thanks and welcome aboard :)
Hello,
we setup a new centos mirror and want you to include it on the list
HTTP: http://centos.mirrors.machost.co
FTP: no
RSYNC: no
Sync schedule: Every 2 hrs
Bandwidth: 10gbps unmetered
Location: US:IL (Chicago)
Sponsor: Mach Host
Sponsor URL: machost.co
IPv4 address to authorize: 198.255.115.92
IPv6 address to authorize: 2001:49f0:d064:10::2
Email contact: info(a)machost.co
Mirroring AltArch : no
thanks.
Hello,
We NFOrce Entertainment B.V. have enabled IPv6 on the mirror for a while but noticed this was not whitelisted in the rsync.
HTTP: http://mirror.nforce.com/CentOS/
FTP: ftp://mirror.nforce.com/CentOS/
RSYNC: rsync://mirror.nforce.com/CentOS/
Sync schedule: Every 4 hrs
Bandwidth: 10GBPS
Location: Amsterdam,Netherlands
Sponsor: NFOrce Entertainment B.V.
Sponsor URL: http://www.nforce.com/
IP to authorize: 85.159.239.121 , 2a00:1768:1000:22:1::1
Email contact: noc(a)nforce.com
I will force IPv4 for now in the rsync. Thank you.
Best Regards,
JeanPaul van der Mijle
NOC Engineer
NFOrce Entertainment BV
Address:
NFOrce Entertainment B.V.
Postbus 1142, 4700 BC
The Netherlands
Telephone:
Voice +31 20 6919299
Fax +31 20 6919409
Hello all,
I have set up a mirror for public usage, details follow. Initial synchronize has been completed.
HTTP: http://mirror.lps.rocks/centos/ <http://mirror.lps.rocks/centos/> https://mirror.lps.rocks/centos/ <https://mirror.lps.rocks/centos/>
FTP: ftp://mirror.lps.rocks/centos/ <ftp://mirror.lps.rocks/centos/> ftps://mirror.lps.rocks/centos/ <ftps://mirror.lps.rocks/centos/>
RSYNC: rsync://mirror.lps.rocks/centos/ <rsync://mirror.lps.rocks/centos/>
Sync Schedule: every 6 hours
Bandwidth: 1Gbps
Location: US, TX, Dallas
Sponsor: LPS
Sponsor URL: https://lps.rocks <https://lps.rocks/>
IPv4 address to authorize: 104.149.238.196
IPv6 address to authorize: 2001:470:1f0f:7d1:104:149:238:196
E-Mail Contact: admin(a)lps.rocks <mailto:admin@lps.rocks>
Mirroring AltArch: No
Thanks!
Hi,
I found some error on my script. Now it's working fine.
Please add my mirror for public listening.
Thank you.
Best regards,
Fahim
On Sun, Feb 19, 2017 at 3:37 AM, Anssi Johansson <centos(a)miuku.net> wrote:
> 18.2.2017, 23.20, Fahim A. kirjoitti:
>
>> Hi,
>>
>> I don't know what happened: I can't see my mirror in the list anymore.
>> also last time it sync, it was 14th February 2017. Please let me know if
>> there is anything wrong.
>>
>
> The problem is the last sync time, 14th February 2017.
> http://mirror-status.centos.org/ lists your mirror as "old" (red). Out of
> date mirrors are not shown on https://www.centos.org/download/mirrors/
> and neither are they shown in the output of mirrorlist.centos.org.
>
> Please make sure you have set up appropriate scripts so that your mirror
> tries to sync periodically, perhaps every six hours or so. Usually this is
> done with cron.
>
> You can also try syncing manually to see if there are problems with the
> sync. If it outputs errors and you are unable to figure out the reason,
> please post the output of your rsync command so that we can help you
> diagnose the problem.
>
> Sometimes the rsync process gets stuck for some reason. Check if you have
> rsync processes running on the mirror, and kill them if necessary.
>
> Once the problem with the sync is corrected, the mirror checking scripts
> will automatically notice that your mirror is up-to-date again and the
> mirror will be listed again. This process may take a few hours, so please
> have some patience.
> _______________________________________________
> CentOS-mirror mailing list
> CentOS-mirror(a)centos.org
> https://lists.centos.org/mailman/listinfo/centos-mirror
>
Hi,
I will be decommissioning my CentOS mirror (centos.trisect.eu) in the
next few weeks. Is this the right place to make this announcement or is
there somewhere else I should inform someone?
Regards
Danny Horne
HTTP: http://mirror.tenten.vn/centos/
HTTP: http://epel.tenten.vn/centos/
Sync schedule: 00:00 (daily)
Bandwidth: 1 Gbps
Location: 6F Sapphire Plaza, 203 Ong Ich Khiem Street, Hai Chau District, Da Nang City, Viet Nam
Sponsor: GMO-Z.com Runsystem
Sponsor URL: tenten.vn
IPv4 address to authorize: all
IPv6 address to authorize: no
Email contact: vutd(a)runsystem.net
Mirroring AltArch : no
________________________________
*IMPORTANT NOTICE*
The information in this e-mail and any attached files is CONFIDENTIAL.
Any unauthorized use or disclosure of this email is prohibited.
If you have received this email in error, please delete the email and its attachments.
GMO-Z.com RUNSYSTEM makes no warranty that this e-mail is error or virus free.
HTTP: http://mirror.neify.es/centos/
FTP: ftp://mirror.neify.es/centos/
RSYNC: rsync://mirror.neify.es/centos/
Sync schedule: Every 6 hrs
Bandwidth: 1GB
Location: Spain, Europe
Sponsor: Neify Hosting
Sponsor URL: https://neify.es
IPv4 address to authorize: 37.59.229.161, 37.59.229.170
Email contact: fernando(a)neify.com
Mirroring AltArch : no
Fernando García
CEO
Neify Hosting SL
Periodista azzati 8
46520 Puerto de Sagunto Valencia
Tlf. +34 902 992 120
Mov +34 667 436 375
Síguenos en - Follow us on:
<http://twitter.com/neify_es> <http://www.facebook.com/NeifyEs>
<http://plus.google.com/100258904023115395892>
https://neify.com <https://neify.com/>
Este mensaje es confidencial y va dirigido únicamente a la persona
propietaria de la dirección de correo.
Si por cualquier error recibe este mensaje, por favor, elimínelo
inmediatamente junto con los documentos adjuntos.
The contens of this email are confidencial. They are intended for the named
recipient only.
If you have received this email by mistake, please delete it immediately and
any attachements.
Thanks for the reply and sorry for the error.
Currently it is in the first level as rsync: //mirror.neify.es/centos/
In my first mail, I added information about my mail client signature (phone,
address ...). Could it be eliminated?
Thank you
Hello,
We have reconfigured and added a rsync service to the server.
Very soon the registrations under the cyberhost.ro domain will became unavailable.
To avoid an failure in the name resolution please change the mirror registration to centos.mirrors.telekom.ro .
Please authorize the server in the public mirror list.
HTTP: http://centos.mirrors.telekom.ro/
FTP: ftp://centos.mirrors.telekom.ro/
RSYNC: rsync://centos.mirrors.telekom.ro/centos/
Sync schedule: Every 6 hrs
Bandwidth: 40Gbps
Location: Romania EU
Sponsor: Telekom Romania
Sponsor URL: www.telekom.ro<http://www.telekom.ro>
IPv4 address to authorize: 92.87.156.3
IPv6 address to authorize: no
Email contact: nicusor.munteanu(a)telekom.ro<mailto:nicusor.munteanu@telekom.ro>
Mirroring AltArch : no
Thank you.
TELEKOM ROMANIA
Nicusor Munteanu
E-mail: nicusor.munteanu(a)telekom.ro<mailto:nicusor.munteanu@telekom.ro>
Web: www.telekom.ro<http://www.telekom.ro/>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Hello!
We'd like to rename our old sub-domain `kodeterbuka.beritagar.id` to
`sumberterbuka.beritagar.id`. Both IPv4 and IPv6 are remain same, only
change the sub-domain.
Thanks in advance.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQI4BAEBCAAiBQJYm1EnGxxkZXdhbmdnYWJhQHh0cmVtZW5pdHJvLm9yZwAKCRDl
f9IgoCjNcJ9ND/9+5nZylRpvLkYE6yL0zhNzsnHLCw2krzHTUdOspN6OQb8nDq/2
S6SGE/wESHwAksQmkJ8w96S5kOQmJonoRAZG0nfJckYXCTR5ChFX+V0vr+EYKXAF
xb0K6E3WUsuFNFUnR/Ely3aG42b1r4poLxZjTnwXV+Uok9P326BJ3ZOSL8yAfwpO
ijy1rnH/nXZTk6uJqsmw+TNIw7tCj6VI3MfuVunms2xcUxw8HQdhyH9pA6101D4z
wFL7O0a5wF3jdl3R7EjmwVuUflYRbiqWYNSQpGI2kiO0ZcaN/jLxoWequn36QQCU
VETD68vuhNoO+7GctgsrcUmBAo8VJD+HVAcGx0UdLxCWvUO902klKYwl5PAiZ7/L
kW29DQ7cdv3JlBLbk0EJq4EQN9FmM/A88E5uBp91j+vMS7TAlGa8NVgsRJWYTZxX
vuavWSZ6BUDAhA8a/IMxNKqBGqlWQmIasz2+x6N+7765JIjLFmmaY3MkrW5ePcdu
xNeCY6+jTDLah32F9nsJmfMjhmNwJj+KPjL9tWWXyFwGWDYJXaxdA58MsZiyScBF
ce0dLdC3mY7iVjmDCRpLFf6vIrUo29tNaluBXgkuIku+TY2NmYTVFljhEWOkIZgn
fRHsLF/qyUftHJtabCCQyluxqnOFSt+CH/G3R18OjGED324dsINfir1gkg==
=lIcf
-----END PGP SIGNATURE-----
Good day,
I'm looking for a resolution to a small issue with the cloud image
repository for the latest CentOS-7 image.
As far as I understand, the latest release of the CentOS-7 image should
always be posted to
http://cloud.centos.org/centos/7/images/CentOS-7-x86_64-GenericCloud.qcow2
The checksum of this file should match the highest numbered version of the
cloud image listed in http://cloud.centos.org/centos/7/images/sha256sum.txt
However, the actual checksum for CentOS-7-x86_64-GenericCloud.qcow2 is
returning as
b56ed1a3a489733d3ff91aca2011f8720c0540b9aa27e46dd0b4f575318dd1fa which
matches for CentOS-7-x86_64-GenericCloud-1608.qcow2 but NOT
CentOS-7-x86_64-GenericCloud-1612.qcow2.
The latest image available should be CentOS-7-x86_64-GenericCloud-1612.qcow2
with a checksum of
559bebb4abb2fdf3bd88c917cc11efb1975bfe4494e666d133a3148ec4e6225d.
To further the confusion, the 'Last modified' image is
CentOS-7-x86_64-GenericCloud-1611.qcow2
and not -1612.
I picked up on this discrepancy because I have a script that monitors &
updates our openstack images when new images become available for various
distributions.
If someone could please correct these inconsistencies it would be greatly
appreciated.
Thanks,
Jared Baker
Hi,
We are pleased to announce new CentOS Public Mirror and would like it to
be listed on CentOS mirrors page.
Mirror Data:
#------------------------------------
HTTP: http://mirror.upb.edu.co/centos/
FTP: NO
RSYNC: NO
Sync schedule: Every 6 hrs
Bandwidth: 400Mb
Location: Medellin, Antioquia, Colombia
Sponsor: Universidad Pontificia Bolivariana
Sponsor URL: http://www.upb.edu.co
IPv4 address to authorize: 200.3.145.185
IPv6 address to authorize: 2801:190:0:A010:0:200:50C8:7B09
Email contact: edwin.gomez(a)upb.edu.co
Mirroring AltArch : no
#------------------------------------
Thanks in advance
--
*Edwin Vladimir Gómez Atuesta.*| Arquitecto Infraestructura | Centro de
Tecnologínas de Información y Comunicación | Universidad Pontificia
Bolivariana | Circular 1 No. 70-01, Bloque 6 oficina 606. Medellín,
COLOMBIA | Tel: +57(4)448 83 88 | Ext. 13267
edwin.gomez(a)upb.edu.co <%3Ca%20href=> | www.upb.edu.co
<http://www.upb.edu.co>
*CONFIDENCIALIDAD:*Este mensaje de correo electrónico contiene
información privilegiada y confidencial; si usted no es el destinatario
real del mismo, elimínelo de manera inmediata, reenvíelo a su remitente
o infórmenos de esta inconsistencia al correo
datos.personales(a)upb.edu.co. <%3Ca%20href=>Así mismo, puede solicitar
actualización de sus datos personales o la eliminación de nuestras
listas de distribución a ese mismo correo.
Para ver más sobre nuestra política de privacidad haga clic aquí.
<http://datospersonales.upb.edu.co/>