[CentOS-mirror] New mirror

Fri Oct 23 21:21:46 UTC 2009

I would generally caution saying that, even the largest mirrors don't 
take these numbers lightly

- Centos' archive is 333G of data
- Centos' primary mirroring is 162G of data
- A single release (5.4) uses 32G of disk

Ignoring bandwidth (that's a lot harder to argue what's useful or not, 
though I'd personally say mirrors should start around 100mbps unless 
they are in a country without a mirror and than anything goes), and 
making the assumption that your only mirroring centos you now need ~500G 
of disk space and a machine with 32G - 24G of ram to be able to serve 
this well.

Why the disk space?
	That's obvious

Why the ram?
	This is something everyone forgets, but it basically boils down
	to the minute you have to start going to disk for your working
	set you will lose when faced with a horde of people trying to
	get data.  If you have less memory than your working data set
	(in the case of Centos 5.4 it's 32G) than your going to be
	thrashing against disk, and ultimately no matter how much
	bandwidth you have you will be limited by the speed that you can
	get that data off of disk.

	Now ram has gotten cheaper, thankfully, but I'm just talking
	about Centos on a mirror, most mirrors have more than just
	Centos on them.  Case in point I have:
		- archlinux
		- centos
		- cpan
		- debian
		- fedora
		- fedora-epel
		- gentoo
		- gnu
		- LDP
		- mandriva
		- moblin
		- oldlinux
		- opensuse
		- redhat
		- slackware
		- suse
		- ubuntu

	Want to guess what my *daily* working data set is?  Want to
	guess what my working data set is when a release happens?  Want
	to guess wat my working data set is when *MULTIPLE* releases
	happen simultaneously?  the answer to some of those is: they
	don't make boxes with enough ram to support that yet, or at
	least that are within the wildest budgetary constraints of a
	mirror.

My argument about getting rid of the CD ISOs isn't about the disk space, 
though that's a valid point (good fast disk is expensive - a 450G 15K 
rpm sas drive currently runs $400+ /ea - thats expensive storage), my 
argument about killing the CD ISOs is one from a ram perspective, though 
looking at my disk usage right now I might be making the disk argument 
in the near future:

/dev/md0              5.5T  4.2T  1.3T  77% /home/mirrors

- John 'Warthog9' Hawley

Nuno Vieira - nfsi wrote:
> Hey folks,
> 
> Who don't have enough bandwith nor storage space doesn't deserve to be a Official CentOS mirror.
> 
> My 100,00 EUR.
> 
> cheers,
> ---
> Nuno Vieira
> nfsi telecom, lda.
> 
> [Email]  nuno.vieira at nfsi.pt
> [Phone]  +351 21 114 2315
> [Phone]  +351 21 142 2300
> [Mobile] +351 91 925 5561
> [Fax]    +351 21 114 2301
> [Web]    http://www.nfsi.pt/ 
> 
> ----- Original Message -----
> From: "J.H." <warthog9 at kernel.org>
> To: "Mailing list for CentOS mirrors." <centos-mirror at centos.org>
> Sent: Friday, 23 October, 2009 8:53:35 PM
> Subject: Re: [CentOS-mirror] New mirror
> 
> Jeff Sheltren wrote:
>> On Oct 23, 2009, at 10:22 AM, Nick Olsen wrote:
>>
>>> Never Thought of that....
>>> I guess your right.
>>> Don't really see why ISO's shouldn't be carried though.
>> Disk space.
>>
>> Some people (I won't name names, *cough* warthog *cough*) might argue  
>> that having ISO images is simply a replication of the packages we're  
>> already carrying on the mirror and that there should be a better way  
>> to handle stuff so that mirrors don't end up with multiple copies of  
>> what is essentially the same data.
> 
> I'm trying my best to kill those stupid ISO images - I mean I've got 
> boot.kernel.org and I've done several installs / upgrades that way 
> (including Centos I might add!), and as a general goal I want to 
> eliminate as many needs to burn a cd for a task as I can.
> 
> That said I realize that I'm "not normal" and at best 5 years ahead of 
> the big curve.  Many people's internet connections are not as good as 
> mine, and it's only 16mbps down / 2mbps up.  Compare that to some of the 
> other places on the planet with 10mbps symetric to 1000mbps symetric and 
> mine pales.
> 
> I have no real expectation however that we will get rid of the ISOs 
> anytime soon.  I would *LOVE* if we could drop the CD ISOs completely 
> from everything, but there's apparently a major backlash every time that 
> happens (Fedora's done it a couple of times now).  I'm kinda hoping that 
> with boot.kernel.org and the DVD ISOs we might be able to finally kill 
> the CD ISO itself off and save all of that space and eliminate that from 
> the possible working set of data.  Just my $0.02 though.
> 
> - John 'Warthog9' Hawley
> _______________________________________________
> CentOS-mirror mailing list
> CentOS-mirror at centos.org
> http://lists.centos.org/mailman/listinfo/centos-mirror
> _______________________________________________
> CentOS-mirror mailing list
> CentOS-mirror at centos.org
> http://lists.centos.org/mailman/listinfo/centos-mirror