On 01/03/17 14:44, Anssi Johansson wrote:
We have 600+ active CentOS mirrors[1] and sometimes there are critical updates that would need to get published to the mirror network as quickly as possible. As of now, it takes around five hours to go from 0 up-to-date mirrors to 80% up-to-date mirrors, with a longer tail for the remaining 20%.
Current guidelines for setting up a mirror[2] specify that mirrors should sync 2-4 times per day. Instead of telling mirrors to sync hourly I thought we could come up with something smarter.
One option that I have considered would be something similar to what the ClamAV guys use to signal end users that there is new content. They use a DNS TXT record for that purpose. For example, as of this writing "dig txt +short current.cvd.clamav.net" produces "0.99.2:57:23148:1488371340:1:63:45637:290", which shows the version numbers for ClamAV itself, main virus database, daily virus database, timestamp and other version numbers.
We could have something similar, showing the timestamps when the content for CentOS, CentOS AltArch and CentOS Vault was last modified, like "1488372781:1487767981:1488113581". "Last modified" in the sense that new packages got added at that time. The idea is that mirrors could set up scripts to check that timestamp from DNS more frequently (such as hourly) without causing load issues to msync nodes by rsyncing hourly. The TTL for the TXT record could be relatively small, like 10 minutes.
As you're all aware, DNS is a prime example of a very scalable system, and that's why I'm fond of this solution. Another option would be to publish the same data in a central location and served over http(s), if relaying the timestamp data via DNS is not desired for some reason.
The basic principle would be "if timestamp in TXT record > my current timestamp (TIME file), synchronize the mirror". With more frequent syncs, mirror admins would need to take care that no two rsync runs would happen at the same time. Using lockfile in the scripts would help with this. I hope that many of the mirror admins already use lockfiles, but providing an example script might help for the newer mirror admins.
The timestamps should be updated only after it has been verified that all (or at least the majority) of msync nodes actually have the content. It takes a while for the data to reach all the msync nodes from the master.
On the other hand, this may cause some traffic peaks for the msync nodes. I don't know how well they would handle the peaks. One obvious way to alleviate the peaks would be to instruct mirror admins to pick a random minute when to check for new content. echo $[ $RANDOM % 60 ] works nicely for this. I don't have statistics, but I believe there might be mirrors that sync at "0 */6" ie. at the top of the hour.
If the traffic peaks to msync nodes is deemed to be a problem, there might be ways to reduce the load to msync nodes. The following idea could be implemented separately from the above timestamp idea, if needed.
There could be some sort of a "web service" which instructs mirrors where to sync from. The core idea in this is that the source might not always be a msync.centos.org server, but it could also be a nearby public mirror that offers rsync and has been verified to have the new content. If requested from Finland, that service could say "ok, you're from .fi, go sync from ftp.funet.fi as it has the new content already" or "uh oh, no nearby external mirrors have the new content, please rsync from eu-msync.centos.org". It could simply return a list of rsync servers in descending priority, with some msync.centos.org addresses at the bottom as fallback. Once the mirror has rsynced, the mirror could ping back and say "I have the content now, please check, and if OK, add me to the list of mirrors that have the new content".
One concern is that the list of rsync sources would need to be protected, so that mirrors could not be tricked into syncing from a malicious source (think DNS poisoning). Ways to protect from this include DNSSEC, TLS and PGP signed data.
Any thoughts about this?
[1] http://mirror-status.centos.org/ [2] https://wiki.centos.org/HowTos/CreatePublicMirrors
Hi Anssi,
Thanks a lot for all those ideas. While I understand your idea behind a TXT record at the DNS level, I'd say that I'm not a fan of that idea. Even with lower TTL, you'd be surprized how many hits we had on servers that were migrated to new ones, as it seems some ISPs aren't obeying the TTL and so were still serving wrong (and expired) A/AAAA records from their cache. DNS itself (currently for centos.org) isn't DNSSEC enabled too, so that would mean other protection, etc
I'd prefer your alternative with "let's host this behind a https web server" and also for the reason that it's easier to have TLS for webserver, and that from the automation point-of-view, it's easier for people allowed to build/sign/push (two people) to just update/drop a file somewhere, than using DNS modification. For dns, as the zone is actually under git/puppet control, that would mean *not* using that, but rather having a delegated zone that would allow nsupdate with a key that those people would share, etc ... So the simple file served from https seems easier from my side.
I'd like to get opinions from Johnny/KB (people able to sign/push) as they'd be directly concerned by that decision.
From an "external mirror admin" PoV, we should also use the centos-mirror list to discuss this, to get their opinions ?
Also, we can divide your proposal into two parts : - external mirrors can check a file they can compare against to sync "faster" than through their cron jobs (discussed above) - modifying completely the msync.centos.org network to have external mirrors not syncing from us, but betwen them (not sure how people feel about this)
PS : Anssi is now part of the mirrors managers team for CentOS , for people not yet aware of that fact
I just wanted to make a note that I have worked out a system which enables me to mirror all of the Fedora repository, which consists of about 12TB in I believe eleven million files, with a polling interval of ten minutes. A typical update (when there are changes to mirror) including the mirrormanager checkin takes about four minutes (most of it waiting for mirrormanager). A poll when there are no changes takes six seconds. The load on the server during a poll is rsync startup time and a handful of stat calls. A full tree traversal on the server is not required. (It may still be required on the client, but that's no worse than plain rsync.)
The software which handles this is at https://pagure.io/quick-fedora-mirror
It involves a server-side component (written in python2 with limited dependencies) to generate file lists in a useful format, and a client side component (currently written in zsh) which fetches the file lists, processes them, calls rsync with a list of changed files, and does a mirrormanager checkin. (The mirrormanager client is not required.)
A tiered setup (mirrors pulling from other mirrors) works fine; only the master mirror ever needs to generate the file lists. None of this limits the ability of clients to mirror in any other way.
Hardlinks are copied as hardlinks assuming that the file lists for all cross-linked rsync modules are regenerated at the same time, and when that doesn't happen, there's an included client-side hardlinker which uses the file list data to more quickly hardlink a repostory.
A form of exclude lists is supported in the client.
We're also working on a no-polling setup using the Fedora message bus, with mirrors automatically waking up and fetching when new content is pushed out.
While the default configurations, the above message bus stuff, and maybe the mirrormanager checkin are Fedora-specific, I do believe the software will work for any rsync server willing to run the file list generator. If any of this interests your project, please let me know.
-
Fabian Arrotin -
Jason L Tibbitts III