I sync from ESnet's Mirror, And with dvd's I'm at 138GB, I've seen it up to 160Gb, I'd allocate it 200GB.

On 4/13/2010 6:01 PM, ^PNNL Public Software Mirror wrote:

Good afternoon,

How much disk space should we allocate a virtual machine to host a full CentOS mirror?

Background:

We're in the process of establishing a new publicly available open source software mirror here at the Pacific Northwest National Laboratory (http://www.pnl.gov). The base system is running Varnish (http://varnish-cache.org/) with 32GB of RAM cache and a 10Gbit/sec network uplink. PNNL has multiple 10Gbit/sec links to ESnet (http://www.es.net/) and the Pacific Northwest GigaPOP (http://www.pnw-gigapop.net/).

For each Linux distribution hosted we're setting up a virtual machine running that distribution. That way we can ensure the distribution's native mirroring and synchronization tools are available.

Internet users will ask the Varnish cache for files, and if a given file isn't in cache then Varnish will pull that file from the back-end virtual machine to serve it to the end user. When a new release hits the street, the most popular CD and DVD ISO images should stay resident in the Varnish RAM cache. We hope this strategy lets us get away with relatively inexpensive / high capacity disk drives while still achieving good network throughput.

I looked through the FAQ and the centos-mirror list archives; if I missed a disk space recommendation please feel free to redirect me and accept my apology for wasting your time.

Thank you and best regards,

Bill Nickless

(for the PNNL Public Software Mirror team) _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror

Your looking at:

137G to cover what's in the main distribution tree

and an optional 393G if you wanted to include the Archives.

If I was to rig out a machine I'd go for a minimum of 250G or so of reasonably fast disk and a fair chunk of ram for CentOS.

As for running each mirror in it's own virtual machine running that OS, this seems like pretty drastic overkill. There are a number of mirrors that don't run the same distro/os as what they are providing, and they have no problems dealing with the mirroring and synchronization (which is mainly handled by rsync, and a cron job). The biggest performance problem a mirror faces is dealing with I/O and keeping that as fast as possible, a Virtual Machine's overhead seems to me to be adding a lot of extra steps in the I/O path that can dramatically slow things down. You may be better served by choosing a single distro that you tune well and not bothering with the VMs.

That's just my $0.02us though and depending on how much bandwidth your looking to dedicate to this the extra VM overhead may be a moot point.

- John 'Warthog9' Hawley

On 04/13/2010 03:01 PM, ^PNNL Public Software Mirror wrote:

Good afternoon,

How much disk space should we allocate a virtual machine to host a full CentOS mirror?

Background:

We're in the process of establishing a new publicly available open source software mirror here at the Pacific Northwest National Laboratory (http://www.pnl.gov). The base system is running Varnish (http://varnish-cache.org/) with 32GB of RAM cache and a 10Gbit/sec network uplink. PNNL has multiple 10Gbit/sec links to ESnet (http://www.es.net/) and the Pacific Northwest GigaPOP (http://www.pnw-gigapop.net/).

For each Linux distribution hosted we're setting up a virtual machine running that distribution. That way we can ensure the distribution's native mirroring and synchronization tools are available.

Internet users will ask the Varnish cache for files, and if a given file isn't in cache then Varnish will pull that file from the back-end virtual machine to serve it to the end user. When a new release hits the street, the most popular CD and DVD ISO images should stay resident in the Varnish RAM cache. We hope this strategy lets us get away with relatively inexpensive / high capacity disk drives while still achieving good network throughput.

I looked through the FAQ and the centos-mirror list archives; if I missed a disk space recommendation please feel free to redirect me and accept my apology for wasting your time.

Thank you and best regards,

Bill Nickless

(for the PNNL Public Software Mirror team) _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror