You're probably looking for something along the lines of mod_cband, calculate based on 200GB per Mbps at 95% and allocate accordingly -- it'll allow clients to burst for small-ish files but fairly effectively ratelimits against 'big' transfers, and can enforce a global cap on transfer per day, week, month :)

I failed to consider that you may be using FTP and/or rsync too ;) Implement something with `tc` to QoS the traffic to a certain rate, and that'll also allow you to handle small/short bursts.

Long-term I consider "Mirror is a little slow for big transfers" better than "Mirror only works for 10 days of each month, then returns errors."

Alex Howells wrote:

You're probably looking for something along the lines of mod_cband, calculate based on 200GB per Mbps at 95% and allocate accordingly -- it'll allow clients to burst for small-ish files but fairly effectively ratelimits against 'big' transfers, and can enforce a global cap on transfer per day, week, month :)

please do NOT do cband controls for centos mirrors, we would prefer to not have a mirror than have something that is forcing users to slow downloads. The way in which mirror urls are used by yum are randomised to spread the load, however a client normally sticks to one mirror for an entire session.

If you are using mod_cband to slow transfers down, you are essentially causing a user to run their transactions slower than what they would have had they moved to another mirror. And at that stage, you may as well just stop offering a mirror.

Personally I'm using the tc route to cap all off campus transfers to 150Mb/s

-------script I use------- #!/bin/bash

tc qdisc add dev eth0 root handle 1: htb default 10

tc class add dev eth0 parent 1: classid 1:90 htb rate 150000Kbit

tc qdisc add dev eth0 parent 1:90 handle 90: sfq perturb 10

#First mark EVERYONE iptables -t mangle -A OUTPUT -j MARK --set-mark 9

#now unmark who we "love" iptables -t mangle -A OUTPUT -d 129.21.0.0/16 -j MARK --set-mark 0

tc filter add dev eth0 protocol ip parent 1:0 handle 9 fw flowid 1:90 ---------/script-------

I used to have a "! 129.21/16" rule but it was too hard to add exceptions to. That is why I now mark everyone and then uncap others. There was a really great howto I think on the gentoo wiki.

-paul On Fri, Apr 25, 2008 at 7:08 PM, Lauro, John jlauro@umflint.edu wrote:

Dropping the DVDs would probably be more noticeable than the patches.

From: centos-mirror-bounces@centos.org [mailto:centos-mirror-bounces@centos.org] On Behalf Of greg@raystedman.org Sent: Friday, April 25, 2008 6:13 PM To: centos-mirror@centos.org Subject: [CentOS-mirror] RayStedman.org Bandwidth

Good Afternoon,

We have excess bandwidth that we have been donating to the CentOS.org mirror for several months now. We are starting to exceed the available bandwidth we have available on our server. I performed some traffic analysis and find that on the 17th of April there was a significant increase in daily traffic from the mirror. Our bandwidth consumption increased from 25GB per day to over 150GB per day and has been holding steady at this level. Our monthly allocation of bandwidth will be exceeded if we continue at this rate.

We would like to continue donating bandwidth to CentOS.org but we need to find a way to control the amount of bandwidth that it is consistent with our monthly allocation. We currently support all releases and the DVDs. Is the correct answer to reduce the amount of content we support on the mirror? Perhaps we should support release 5.x only?

Please give me your thoughts on how to best remain below our monthly bandwidth allocation and support the CentOS.org community at the same time.

Have a great day everyone! Greg

---

CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror