Hi,
At the end of the Dojo in Brussels, I had the chance to field the question to our contributor audience : how can we get security updates out to the user machines faster.
At the moment, things are setup like any other distro or large open source content network is : we rsync in stages, and external mirrors pickup every 4 to 6 hours, some external mirrors pickup from other external mirrors. Net result is that for a given update, it can be upto 16 to 18 hours before we get a majority content sync in front of most users.
In cases like the recent Glibc issue, 18 hrs can be a long time since the release ( remember, we already lag RHEL releases since our process starts once theirs ends ).
There were a couple of ideas that came up in the conversation at the Dojo, and then in the following conversations over the entire Fosdem weekend. The two that seemed most likely, easiest to implement and perhaps most robust, involved a chunk of the load moving to mirror.centos.org for some period of time. These are :
A) we setup a rapid update repo, that would be hosted on and run from mirror.centos.org exclusively. The yum repo definitions would still point at mirrorlist, however they would only expect centos.org urls in the baseurl stack from mirrorlist.centos.org; This would allow us to reduce the overall to-user-visibility in default centos linux installs to under an hour for content upto 250MB in size.
B) integrate the mirrorlist backend with the release mechanism in centos linux, so when there is a new updates pushed, all updates are then delivered via mirror.centos.org for the next 24 hrs. After this period, traffic reshapes to be delivered from the external mirrors by default.
The Key issue to note with (A) is that while we might push something to this rapid update repo, the same content will also be available in the regular updates/ repo. So once its starts showing up externally, traffic will naturally switch to using the updates repo from local mirrors ( using repo names and cost etc, we can influence repo priority where there is common content ).
The second important thing to note here is traffic capacity : we can deliver upto 20 Gbit/sec in the US and EU from centos.org - and around 8 Gbit/sec for everywhere else. Given that we prefer to offload traffic to external mirrors at the moment, its hard to estimate what the overall capacity requirements will be. Ideally, drpms should/would help.
Depending on how everyone feels around this, I'd like to go ahead and kick off implementation around this. Lets get it done before the next big security update comes through.
Regards