mirror

mirror@lists.centos.org

4577 discussions

big_bw: Bandwidth Reporting Script
by greg＠raystedman.org 02 May '08

02 May '08

I vote for Gilbert's version of this script!! His version is much, much faster in execution as everything is done in memory. Now it is time for me to study up on the use of awk and associative arrays. Thanks Gilbert! Greg

2 1

big_bw: Bandwidth Reporting Script
by greg＠raystedman.org 01 May '08

01 May '08

Good Morning, The discussion about "RayStedman.org Bandwidth" inspired me to write a script that reports who the largest bandwidth consumers by ip address and host name. The report looks like this: 3,867,534,553 66.159.202.142 adsl-66-159-202-142.dslextreme.com. 3,847,010,060 190.82.182.19 190-82-182-19.adsl.cust.tie.cl. 1,410,308,739 130.160.110.250 1,051,088,947 216.57.200.57 I'm sure this kind of thing has been done many times in the past by other tools. I thought I would post the script I created just in case it might be helpful to others on this forum. Thanks again for your feedback on this topic. Greg #!/bin/bash # big_bw -- written by Greg Sims 05/01/08 # this script takes as input apache httpd log files access_log and # access_log.processed. a report is generated that contains one line # per ip address with the following fields: bandwidth consumed, # the ip address and the host name associated with the ip address. # # it is important to use mod_logio in the creation of the log files # to ensure the proper number of bytes are recorded in each log # entry. please see http://www.devside.net/guides/config/bytes-sent # how to accomplish this. # directory where access_log and access_log.processed are located # basedir="/var/www/vhosts/raystedman.net/statistics/logs/" # create bw.raw containing the ip address and bandwidth for each record; # sort the resulting file by ip address # cd /tmp cat $basedir"access_log" >bw.log cat $basedir"access_log.processed" >>bw.log cat bw.log | cut -d' ' --field=1,10 | sort >bw.raw # read through bw.raw and create bw.sum which contains one line per # ip address. each line in bw.sum contains the amount of bandwidth # consumed and the ip address that used the bandwidth # thisip="" rm -f bw.sum while read inputline; do ip=$(echo "$inputline" | cut -d " " -f 1) bw=$(echo "$inputline" | cut -d " " -f 2) if [ "$bw" = "-" ]; then bw=0 fi if [ "$thisip" != "$ip" ]; then echo $thisipbw $thisip >>bw.sum thisip=$ip thisipbw=$bw else if [ $bw != "-" ]; then thisipbw=$(( $thisipbw + $bw )) fi fi done < "bw.raw" # sort bw.sum so the largest amount of bandwidth used is at the top. # create bw.sum.sort which is the largest 35 consumers of bandwidth. # write a report to stdout doing some formatting in the process. # sort -nr bw.sum | head -n 35 >bw.sum.sort while read inputline; do bw=$(echo "$inputline" | cut -d " " -f 1) bw=$(echo "$bw" | sed -e :a -e 's/$.*[0-9]$$[0-9]\{3\}$/\1,\2/;ta') ip=$(echo "$inputline" | cut -d " " -f 2) echo -n $bw | sed -e :a -e 's/^.\{1,14\}$/ &/;ta' echo -n " " echo -n $ip | sed -e :a -e 's/^.\{1,15\}$/ &/;ta' echo -n " " host_name=$(host $ip | sed 's/^.*pointer //' | sed 's/.*DOMAIN)//') host_name=$(echo "$host_name" | sed 's/.*alias for //') echo $host_name done <"bw.sum.sort"

2 1

New Canadian Tier 2 mirror site
by Gilbert E. Detillieux 28 Apr '08

28 Apr '08

Country (Region) CA (Canada) State (Area) MB City Winnipeg Organization Name Manitoba Unix User Group http://www.muug.mb.ca/ Versions 5 Architectures i386 x86_64 Direct DVD Downloads (depends on upstream mirror)** URL's: HTTP http://www.muug.mb.ca/pub/centos/ FTP ftp://ftp.muug.mb.ca/pub/centos/ RSYNC rsync://rsync.muug.mb.ca/centos/ Bandwidth cap 100 MB/s on I2/CA*Net4 (possibly lower on Commercial link, due to traffic shaping) Preferred contact www(a)muug.mb.ca Synch. frequency 4 times daily: 5*/updates once daily: All ** Note, we're currently mirroring from Argonne National Laboratory (mirror.anl.gov), over Internet 2, since we're on the Canadian equivalent (CA*Net 4) which connects to I2 over Gigabit links. According to the CentOS Public Mirrors page, they don't allow direct DVD downloads, yet they do have them available via rsync (in the 5.1/isos/* directories). As long as we can get them from ANL, we'll allow direct downloads of them from our site too. If we should be pulling from a different mirror site, please let us know. -- Gilbert E. Detillieux E-mail: <gedetil(a)muug.mb.ca> Manitoba UNIX User Group Web: http://www.muug.mb.ca/ PO Box 130 St-Boniface Phone: (204)474-8161 Winnipeg MB CANADA R2H 3B4 Fax: (204)474-7609

2 1

RayStedman.org Bandwidth
by greg＠raystedman.org 28 Apr '08

28 Apr '08

Thank you very much for the feedback here guys! I believe we will drop the DVDs for now and see where that leaves us. Tru: Please change us to DVD=No. We also have some bandwidth and connection limiting that is available under Plesk that we might be able to use. It does not look like Plesk will allow us to limit resource on a subdomain basis so we would likely need to use RayStedman.net instead of RayStedman.org for hosting the CentOS Mirror. This would also allow us to separate out our traffic stats from the main website a bit easier. Mod_cband looks like it has a great deal of flexibility as well. I will try to read up on it over the next few days. Does anyone know if you can specify a subdomain as a limiting target? We have several options to work with now. Thanks again! Greg

3 3

CentOS mirror late http://mirror.exetel.com.au/pub/centos/
by Tru Huynh 28 Apr '08

28 Apr '08

Hi, According to http://mirror-status.centos.org/ http://mirror.exetel.com.au/pub/centos/ has started lagging behind (3.6 days). any issue? Best regards, and thanks for providing a public CentOS mirror. Tru -- Tru Huynh (mirrors, CentOS-3 i386/x86_64 Package Maintenance) http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBEFA581B

1 1

RayStedman.org Bandwidth
by greg＠raystedman.org 28 Apr '08

28 Apr '08

Hey All, The comments you folks have made on this subject have really been great. We are taking two steps here to help address our bandwidth limitations: (1) Isolate CentOS traffic to a unique domain and ip address. This will help us monitor the traffic from the mirror separate from our main domain. This will also give us additional flexibility for how we support the CentOS mirror in the future. I will look at the ip addresses requesting transfers and see if there is a pattern of ownership. ==>> Please change the domain for the mirror from RayStedman.org to RayStedman.net. DNS propagation is in progress as I type this append. (2) We are going to eliminate support for DVDs for all releases. ==>> Please update our mirror to DVD=No. I hope the combination of these two changes will resolve this issue. Thank you again for your comments! Greg

2 1

RayStedman.org Bandwidth
by greg＠raystedman.org 27 Apr '08

27 Apr '08

Good Afternoon, We have excess bandwidth that we have been donating to the CentOS.org mirror for several months now. We are starting to exceed the available bandwidth we have available on our server. I performed some traffic analysis and find that on the 17th of April there was a significant increase in daily traffic from the mirror. Our bandwidth consumption increased from 25GB per day to over 150GB per day and has been holding steady at this level. Our monthly allocation of bandwidth will be exceeded if we continue at this rate. We would like to continue donating bandwidth to CentOS.org but we need to find a way to control the amount of bandwidth that it is consistent with our monthly allocation. We currently support all releases and the DVDs. Is the correct answer to reduce the amount of content we support on the mirror? Perhaps we should support release 5.x only? Please give me your thoughts on how to best remain below our monthly bandwidth allocation and support the CentOS.org community at the same time. Have a great day everyone! Greg

5 6

New Mirror centos.lqdinternet.com
by LQD Internet Admin 24 Apr '08

24 Apr '08

Provider Name : LQD Internet Limited Provider URL : HYPERLINK "http://www.lqdinternet.com/"http://www.lqdinternet.com/ http : HYPERLINK "http://centos.lqdinternet.com/"http://centos.lqdinternet.com/ Content : All Arch : All sync rate : 4 times per day from "us-msync.centos.org" Bandwidth : 100Mbps connection / unlimited traffic Location : Orlando, FL, USA Admin : admin [at] lqdinternet.com Anonymous ftp coming soon. No virus found in this outgoing message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.23.4/1394 - Release Date: 23/04/2008 19:16

2 1

How do I restrict our mirror to English only files?
by Patrick.Young＠chghealthcare.com 23 Apr '08

23 Apr '08

Hi Everyone, I'm setting up an in-house mirror for our company that will not be available to the outside world. To save disk space and bandwidth usage I'd like to mirror only the English versions of files rather than all languages available. Does anyone know an efficient way to do this? Is there an rsync command to do this? Thanks for your help, Patrick

2 1

New Mirror
by Corey Baldwin 23 Apr '08

23 Apr '08

Provider Name : Mojohost Provider URL : http://www.mojohost.com/ http : http://mirror.mojohost.com/centos ftp : ftp://mirror.mojohost.com/centos Content : All Arch : All sync rate : 4 times per day from "msync.centos.org" Bandwidth : 20 Mbps Location : Miami, FL, USA Admin : net-admin at mojohost.com -- Thank you, Corey Baldwin Mojohost..

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

mirror