On Fri, Nov 20, 2009 at 9:36 AM, Clarkson University Mirror Admin < mirror-admin@cslabs.clarkson.edu> wrote:

On Thu, Nov 19, 2009 at 6:50 PM, Bob Bownes bownes@gmail.com wrote:

...

Anyone else seeing high numbers of requests for the DVD isos from a few discrete locations? I'm getting multiple requests for dvd's from over 500 separate locations.

Top 10 offenders:

5502 59.37.17.20 6616 123.127.231.205 6662 122.205.13.1 6993 218.69.255.86 7137 210.22.151.90 7648 p108.net059086006.tnc.ne.jp 8809 114.92.117.186 10262 210.72.27.62 11409 114.255.44.131 13682 221.10.84.188

Been going on for about the last 24 hours.

Thanks, Bob

Bob,

Nice catch. I'm seeing some of the same IPs (listed below) on my mirror as well. I'm not sure whether I should block them or not though because I'd prefer not block people actually getting the ISOs but I also don't need bandwidth being consumed for no good reason since my mirror is already maxing out it's caps.

Matt

6272 122.205.13.1 http 206 1519 114.92.117.186 200 4896 210.72.27.62 200 2192 114.255.44.131 9970 221.10.84.188

-- Mathew S. McCarrell Clarkson University '10

mccarrms@gmail.com mccarrms@clarkson.edu 1-518-314-9214

Hi!

I too am being hit, hard.. Although I did not carry DVD iso but the quasi-DOS on the CD iso files are just as bad. Not just bandwidth consumption but several dozens simultaneous connections from a single IP. I think it's due to clients behind these IPs using some kind of download managers.

So much so it is affecting those others who are grabbing *.rpm's download performance.

I am tempted to use mod_bw and limit *.iso bandwidth to a reasonable low max download speed and limitipconn to limit those download manager users to a reasonable simultaneous connection.

Regards

Maulvi

A lot of mirrors do this anyways. ISO's are big files, HTTP wasn't made to move big files. Most people do this just so the user has a better chance of the download completing successfully.

On 11/25/2009 6:10 PM, Pär Andersson wrote:

Mogens Kjaermk@crc.dk writes:

...

Redirecting http requests to ftp solved the problem:

Is this really ok on a CentOS mirror? In the mirror list it says that you support HTTP but with this config that is not true for ISOs.

For most people this probably makes no difference, but there are networks where stupid NAT and/or firewalls lets you access HTTP but not FTP.

I limit the amount of connections/IP to 2 for the ISOs instead, this seems to work quite ok against download accelerators.

-- Pär Andersson

---

CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror

To be 100% honest, I was going off the fedora mirroring tutorial, And Was mistaken, It states this.

Redirecting ISO downloads to FTP

Apache 2.x and earlier can't distribute files larger than 2GB. This means DVD images won't work. (lighttpd doesn't have this limitation). Also, some people find FTP to be more efficient than HTTP for really large files like ISO images. These Rewrite lines will redirect all HTTP GET requests for *.iso files to a different FTP daemon. With this method, HEAD requests used by the MirrorManager crawler for *.iso files aren't rewritten, which gives better crawling results.

On 11/25/2009 10:00 PM, H. Peter Anvin wrote:

On 11/25/2009 03:19 PM, Nick Olsen wrote:

...

A lot of mirrors do this anyways. ISO's are big files, HTTP wasn't made to move big files. Most people do this just so the user has a better chance of the download completing successfully.

What makes you say that http was not made to move big files? There is absolutely nothing about http which makes it unsuitable for large files.

I would personally consider a mirror redirecting http traffic to ftp to be broken. It's at the very best rude for people behind firewalls.

-hpa

On 11/26/2009 11:39 AM, Karanbir Singh wrote: ...

We dont want people doing this - which is why we segregate DVD mirrors from the others. If a mirror is doing this, they should only be listed as a ftp mirror and not as someone doing http as well.

Then please remove "HTTP" for ftp.crc.dk (130.226.184.9) from the mirror list.

Mogens

Greetings,

Here's an article I wrote some time ago about this problem. I assume the behavior you guys are seeing matches that in the article.

You may or may not like the solution I used. I think the site that hosted the apache module I used is dead now but if anyone wants the code, it is attached to the article.

mod_cband to the Rescue http://www.montanalinux.org/mod_cband.html

TYL,

On Fri, 4 Dec 2009, Randy McAnally wrote:

mod_limitipconn does the trick, and it's a well maintained/supported module. ...

...

mod_cband to the Rescue http://www.montanalinux.org/mod_cband.html

mod_limitipconn and mod_cband do slightly different things. mod_cband does some of what mod_limitipconn does and also some of what mod_bandwidth does. mod_limitipconn can be used to restrict connections per ip, but from what I can see, it cannot do

(a) bandwidth limiting or throttling which mod_cband can do

(b) set different connections per ip limit for different source ip addresses (or blocks of ip addresses) which also mod_cband can do

Also, mod_cband has a nifty little status page. In addition to the link above, there is a decent howto at

http://www.howtoforge.com/mod_cband_apache2_bandwidth_quota_throttling