[CentOS-devel] setting up an emergency update route

Sat Feb 7 20:44:10 UTC 2015
Stephen John Smoogen <smooge at gmail.com>

On 7 February 2015 at 08:12, Tim Verhoeven <tim.verhoeven.be at gmail.com>
wrote:

> Hi,
>
> I've been thinking a bit about this. The best solution IMHO besides
> building your our CDN, which is indeed a bit over the top for this, is
> to push these updates instead of working with a pull method. So would
> it be possible to find some mirrors that would allow us to push
> packages into our repo's on their servers. In case of releases that
> need to go out quickly we could use a seperate mirrorlist that only
> includes our servers and the mirrors that allows us to push to. So we
> can move the needed packages our quickly and let users get them fast.
> Later as the other mirrors sync up we just go back to the normal
> mirrorlist.
>
> Stupid idea or not?
>
>
I don't think it is "stupid", but it is overly simplified. Just going off
of the EPEL checkins to mirrorlist there are at least 400k->600k active
systems which are going to be checking hourly for updates for an emergency
update. The number of mirrors who are going to allow a push system are
going to have to be large enough to deal with the thundering herd problem
when an update occurs and 500k systems checkin at 10 after the hour (seems
like a common time for boxes which check in hourly) all see there is a new
update and start pulling from it.

In the many years of mirror administration, there have been multiple
requests for some sort of push system to allow for better speedy downloads.
Out of the thousands of mirrors, the number who say they will do it are
usually less than 10. And none of them the guys with very large bandwidth.

Take problem A add it to problem B and you end up with a recipe for
complete meltdown of a service you are hoping to help people better.

Problem A isn't something that anyone can fix. The hundreds of thousands to
millions of systems out there that look for updates regularly aren't
something you can administer to. You can give them premade crontabs, etc
etc and you will find that 10%-15% of the people who were checking in at 10
after the hour now are doing ti around the hour.. but you still have a huge
lump at 10 after the hour. [Mainly because sysadmins like to use the script
they know has worked for the last 10+ years versus some god knows who
tested it script.]

Problem B is one that could possibly dealt with but it is not just
convincing the mirror administrators but their management that this is an
acceptable risk in security, network bandwidth costs, and other factors.
That takes a lot of social capital, marketing and general sales skills. If
you have them, then you have a better chance of accomplishing it than most
system administrators.

-- 
Stephen J Smoogen.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20150207/5e14cfc0/attachment-0008.html>