[CentOS-devel] setting up an emergency update route

[Karanbir Singh]

Hi,

At the end of the Dojo in Brussels, I had the chance to field the
question to our contributor audience : how can we get security updates
out to the user machines faster.

At the moment, things are setup like any other distro or large open
source content network is : we rsync in stages, and external mirrors
pickup every 4 to 6 hours, some external mirrors pickup from other
external mirrors. Net result is that for a given update, it can be upto
16 to 18 hours before we get a majority content sync in front of most users.

In cases like the recent Glibc issue, 18 hrs can be a long time since
the release ( remember, we already lag RHEL releases since our process
starts once theirs ends ).

There were a couple of ideas that came up in the conversation at the
Dojo, and then in the following conversations over the entire Fosdem
weekend. The two that seemed most likely, easiest to implement and
perhaps most robust, involved a chunk of the load moving to
mirror.centos.org for some period of time. These are :

A) we setup a rapid update repo, that would be hosted on and run from
mirror.centos.org exclusively. The yum repo definitions would still
point at mirrorlist, however they would only expect centos.org urls in
the baseurl stack from mirrorlist.centos.org; This would allow us to
reduce the overall to-user-visibility in default centos linux installs
to under an hour for content upto 250MB in size.

B) integrate the mirrorlist backend with the release mechanism in centos
linux, so when there is a new updates pushed, all updates are then
delivered via mirror.centos.org for the next 24 hrs. After this period,
traffic reshapes to be delivered from the external mirrors by default.

The Key issue to note with (A) is that while we might push something to
this rapid update repo, the same content will also be available in the
regular updates/ repo. So once its starts showing up externally, traffic
will naturally switch to using the updates repo from local mirrors (
using repo names and cost etc, we can influence repo priority where
there is common content ).

The second important thing to note here is traffic capacity : we can
deliver upto 20 Gbit/sec in the US and EU  from centos.org - and around
8 Gbit/sec for everywhere else. Given that we prefer to offload traffic
to external mirrors at the moment, its hard to estimate what the overall
capacity requirements will be. Ideally, drpms should/would help.

Depending on how everyone feels around this, I'd like to go ahead and
kick off implementation around this. Lets get it done before the next
big security update comes through.

Regards


-- 
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc