[CentOS-mirror] 5.6 is coming closer

Wed Apr 6 17:37:11 UTC 2011
J.H. <warthog9 at kernel.org>

On 04/05/2011 01:52 PM, Karanbir Singh wrote:
> On 04/05/2011 12:30 PM, Roelf Wichertjes wrote:
>>
>> Maybe a idea,
>> Why not choose the least busy ones,
>> Say there are 100 mirrors, 5 are busy and 5 almost unused
>> Isn't it a better idea to let the 5 busy and the 5 unbusy pull from centos.org
>> And have the other 90 pull from the 5 unbusy?
>> That should even the load better.
>>> On 04/05/2011 10:20 AM, Prof. P. Sriram wrote:
>>>> Maybe it's been discussed before, but would it not be worthwhile to do a
>>>> DNS based thing for this? We create a temporary rsync source domain
>>>
>>> Thats quite a lot of work, I'm more keen on having ACL's in place that
>>> only allow some specific mirrors ( maybe the 100 busiest ones ) to pull
>>> from centos.org; and have everyone else pull from them.
> 
> 
> from 'busy' -i meant more like kernel.org / heanet.ie or mirrorservice.org
> 
> - KB

Tiering the mirror distribution is pretty common, and honestly makes
things a *LOT* easier for everyone.  I agree with the sentiments already
stated, automation is what makes this all doable.  Removing things like
--delete from your mirrors, is just a PITA.  Yes accidental upstream
removals will happen, but if the mirror infrastructure is structured
well it will propagate out and the fix will propagate out quickly.

The way I've normally seen it is a small number (say 10) mirrors are
allowed to pull form the master machines, and servers are then
encouraged / forced to pull from those tier 1 mirrors.  This means the
tier 1's can pull more often from the upstream, and everyone else can
make better use of the 1 & 10 gbps (and associated big hardware) links
some of the bigger mirrors have.  Personally I think it's worthwhile,
and it's not too hard to implement.

Keep in mind that the 'busier' servers (kernel.org at least) are in a
better position (hardware / bandwidth) to support a greater number of
people pulling from them.  I would guess many of those "unused" mirrors
may not be able to support the deluge you could potentially be pointing
at them, this isn't universal but it's something to be aware of.

- John 'Warthog9' Hawley