[CentOS-mirror] Chinese IPs - Mirror Stats

Fri Jan 22 08:43:30 UTC 2010
Prof. P. Sriram <sriram at ae.iitm.ac.in>

On Thu, 21 Jan 2010, Scott Adametz wrote:
> Most of the traffic came from Chinese addresses in the 114.249.219.0,
> 121.41.181.0, 221.0.0.0, 123.118.107.0 and 218.1.7.200.0 subnets.  
> According to GeoIP these originate from Beijing, Zhejiang, Hubei,
> Beijing and Fujian respectively.  Each downloaded approximately 23TB,
> 17TB, 10TB, 10TB, 10TB and exhibited similar repetitive patterns of the
> same file.

We had a similar issue at the centos (and other stuff) mirror at
ftp.iitm.ac.in some months ago. We have solved it effectively using per ip
connection limit and fail2ban. It appears that the traffic originates via
a download accerlator that is popular in china. We used to get the similar
thousands of ranged requests for the iso image files of centos and other
linux distributions. We have put a per-ip connection limit of 5 using the
limitipconn module. Connection attempts over 5 get logged in the apache
error log. fail2ban package is used to monitor this log file; when any
single ip generates more than 5 error message in a minute (meaning that ip
has tried to open more than 5 connections more than 5 times in a minute),
the fail2ban package inserts an iptables firewall rule that blocks ALL
connection requests from this IP for the next one hour. After a few
minutes, the 5 existing (ranged download request) connections complete
their download and the offending IP is locked out for the rest of the
hour. Works very very effectively. We saw our hit rate drop from about
700,000 per day to below 100,000 per day. We continue to server the centos
(and other) mirror community. Scott, I would urge you to seriously
consider this type of solution instead of dropping out of the mirror
network. I will be happy to provide any further assistance in this regard.

-- 
sriram