I'm setting up a public mirror (ftp only so far) on one of our servers and was wondering what kind of bandwidth usage to plan for. We have about 100M overall at this site, but I want to make sure that I can limit the rate appropriately. I had planned on doing this using the ftp server's configuration (vsftpd).
Anyone have any notes on a good ballpark figure for maximum number of connections and maximum bandwidth per connection? Any horror stories, grim warnings or sage advice appreciated.
----- Original Message ----- | I'm setting up a public mirror (ftp only so far) on one of our servers | and was wondering what kind of bandwidth usage to plan for. We have | about 100M overall at this site, but I want to make sure that I can | limit the rate appropriately. I had planned on doing this using the | ftp | server's configuration (vsftpd). | | Anyone have any notes on a good ballpark figure for maximum number of | connections and maximum bandwidth per connection? Any horror stories, | grim warnings or sage advice appreciated.
I was averaging about 50-100MBps when I initially deployed. When the mirror was in full swing I was saturating my Gigabit switches. It's now been throttled to 20MBps during peak hours if you aren't on CA*Net or Canarie and if you are 50MBps during peak hours.
Ours is quite bursty, Sits around only 5-30mb/s normally. But will sometimes hammer the 100Mb/s ethernet port its on for 20minutes to an hour.
On 3/18/2011 12:33 PM, James A. Peltier wrote:
----- Original Message ----- | I'm setting up a public mirror (ftp only so far) on one of our servers | and was wondering what kind of bandwidth usage to plan for. We have | about 100M overall at this site, but I want to make sure that I can | limit the rate appropriately. I had planned on doing this using the | ftp | server's configuration (vsftpd). | | Anyone have any notes on a good ballpark figure for maximum number of | connections and maximum bandwidth per connection? Any horror stories, | grim warnings or sage advice appreciated.
I was averaging about 50-100MBps when I initially deployed. When the mirror was in full swing I was saturating my Gigabit switches. It's now been throttled to 20MBps during peak hours if you aren't on CA*Net or Canarie and if you are 50MBps during peak hours.
So do users have a suggestion on how to throttle / rate limit their mirror server? Mine sits on a gig-e connection, and I just got a call from campus IT questioning the amount of bandwidth I'm using...Right now, I'm running it "fully open", but I may have to restrict that, at least during certain hours. I run http, ftp, and rsync on my server.
Thanks!
--Jim
On Fri, Mar 18, 2011 at 9:35 AM, Nick Olsen Nick@141networks.com wrote:
Ours is quite bursty, Sits around only 5-30mb/s normally. But will sometimes hammer the 100Mb/s ethernet port its on for 20minutes to an hour.
On 3/18/2011 12:33 PM, James A. Peltier wrote:
----- Original Message ----- | I'm setting up a public mirror (ftp only so far) on one of our servers | and was wondering what kind of bandwidth usage to plan for. We have | about 100M overall at this site, but I want to make sure that I can | limit the rate appropriately. I had planned on doing this using the | ftp | server's configuration (vsftpd). | | Anyone have any notes on a good ballpark figure for maximum number of | connections and maximum bandwidth per connection? Any horror stories, | grim warnings or sage advice appreciated.
I was averaging about 50-100MBps when I initially deployed. When the mirror was in full swing I was saturating my Gigabit switches. It's now been throttled to 20MBps during peak hours if you aren't on CA*Net or Canarie and if you are 50MBps during peak hours.
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
I would limit it on the upstream router. We just give ours a 100mb/s port and call it a day. But I think rate limiting upstream would work the best if you need it to run speeds other then just standard eth speeds (10/100/1000).
On 3/22/2011 12:11 PM, Jim Kusznir wrote:
So do users have a suggestion on how to throttle / rate limit their mirror server? Mine sits on a gig-e connection, and I just got a call from campus IT questioning the amount of bandwidth I'm using...Right now, I'm running it "fully open", but I may have to restrict that, at least during certain hours. I run http, ftp, and rsync on my server.
Thanks!
--Jim
On Fri, Mar 18, 2011 at 9:35 AM, Nick OlsenNick@141networks.com wrote:
Ours is quite bursty, Sits around only 5-30mb/s normally. But will sometimes hammer the 100Mb/s ethernet port its on for 20minutes to an hour.
On 3/18/2011 12:33 PM, James A. Peltier wrote:
----- Original Message ----- | I'm setting up a public mirror (ftp only so far) on one of our servers | and was wondering what kind of bandwidth usage to plan for. We have | about 100M overall at this site, but I want to make sure that I can | limit the rate appropriately. I had planned on doing this using the | ftp | server's configuration (vsftpd). | | Anyone have any notes on a good ballpark figure for maximum number of | connections and maximum bandwidth per connection? Any horror stories, | grim warnings or sage advice appreciated.
I was averaging about 50-100MBps when I initially deployed. When the mirror was in full swing I was saturating my Gigabit switches. It's now been throttled to 20MBps during peak hours if you aren't on CA*Net or Canarie and if you are 50MBps during peak hours.
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
Unfortunately, I don't run the upstream router. And I just got shut down by central IT...Apparently I was overloading their traffic shaper and impeding performance campus-wide.
So I need to come up with a rate limiting method that runs on my mirror server. I know linux has QoS and other traffic shaping systems, I just haven't found a very clear guide to using them, and I was hoping for someone using traffic shaping on linux with mirror servers to supply some suggested settings.
--Jim
On Tue, Mar 22, 2011 at 9:13 AM, Nick Olsen Nick@141networks.com wrote:
I would limit it on the upstream router. We just give ours a 100mb/s port and call it a day. But I think rate limiting upstream would work the best if you need it to run speeds other then just standard eth speeds (10/100/1000).
On 3/22/2011 12:11 PM, Jim Kusznir wrote:
So do users have a suggestion on how to throttle / rate limit their mirror server? Mine sits on a gig-e connection, and I just got a call from campus IT questioning the amount of bandwidth I'm using...Right now, I'm running it "fully open", but I may have to restrict that, at least during certain hours. I run http, ftp, and rsync on my server.
Thanks!
--Jim
On Fri, Mar 18, 2011 at 9:35 AM, Nick OlsenNick@141networks.com wrote:
Ours is quite bursty, Sits around only 5-30mb/s normally. But will sometimes hammer the 100Mb/s ethernet port its on for 20minutes to an hour.
On 3/18/2011 12:33 PM, James A. Peltier wrote:
----- Original Message ----- | I'm setting up a public mirror (ftp only so far) on one of our servers | and was wondering what kind of bandwidth usage to plan for. We have | about 100M overall at this site, but I want to make sure that I can | limit the rate appropriately. I had planned on doing this using the | ftp | server's configuration (vsftpd). | | Anyone have any notes on a good ballpark figure for maximum number of | connections and maximum bandwidth per connection? Any horror stories, | grim warnings or sage advice appreciated.
I was averaging about 50-100MBps when I initially deployed. When the mirror was in full swing I was saturating my Gigabit switches. It's now been throttled to 20MBps during peak hours if you aren't on CA*Net or Canarie and if you are 50MBps during peak hours.
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
We only serve via HTTP today - but we run mod_security in order to keep a grasp on traffic levels. CentOS in particular we don't have any limits on as it never gets higher than what we are prepared for, but we mirror *many* other things and there's a few we have to keep an eye on (such as Mozilla mirror) especially during release days.
Paul
-----Original Message----- From: centos-mirror-bounces@centos.org [mailto:centos-mirror-bounces@centos.org] On Behalf Of Jim Kusznir Sent: Tuesday, March 22, 2011 12:24 PM To: Mailing list for CentOS mirrors. Subject: Re: [CentOS-mirror] Bandwidth questions
Unfortunately, I don't run the upstream router. And I just got shut down by central IT...Apparently I was overloading their traffic shaper and impeding performance campus-wide.
So I need to come up with a rate limiting method that runs on my mirror server. I know linux has QoS and other traffic shaping systems, I just haven't found a very clear guide to using them, and I was hoping for someone using traffic shaping on linux with mirror servers to supply some suggested settings.
--Jim
On Tue, Mar 22, 2011 at 9:13 AM, Nick Olsen Nick@141networks.com wrote:
I would limit it on the upstream router. We just give ours a 100mb/s port and call it a day. But I think rate limiting upstream would work the best if you need it to run speeds other then just standard eth speeds (10/100/1000).
On 3/22/2011 12:11 PM, Jim Kusznir wrote:
So do users have a suggestion on how to throttle / rate limit their mirror server? Mine sits on a gig-e connection, and I just got a call from campus IT questioning the amount of bandwidth I'm using...Right now, I'm running it "fully open", but I may have to restrict that, at least during certain hours. I run http, ftp, and rsync on my server.
Thanks!
--Jim
On Fri, Mar 18, 2011 at 9:35 AM, Nick OlsenNick@141networks.com wrote:
Ours is quite bursty, Sits around only 5-30mb/s normally. But will sometimes hammer the 100Mb/s ethernet port its on for 20minutes to an hour.
On 3/18/2011 12:33 PM, James A. Peltier wrote:
----- Original Message ----- | I'm setting up a public mirror (ftp only so far) on one of our servers | and was wondering what kind of bandwidth usage to plan for. We have | about 100M overall at this site, but I want to make sure that I can | limit the rate appropriately. I had planned on doing this using the | ftp | server's configuration (vsftpd). | | Anyone have any notes on a good ballpark figure for maximum number of | connections and maximum bandwidth per connection? Any horror stories, | grim warnings or sage advice appreciated.
I was averaging about 50-100MBps when I initially deployed. When the mirror was in full swing I was saturating my Gigabit switches. It's now been throttled to 20MBps during peak hours if you aren't on CA*Net or Canarie and if you are 50MBps during peak hours.
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
_______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
From: centos-mirror-bounces@centos.org [centos-mirror-bounces@centos.org] on behalf of Jim Kusznir
So do users have a suggestion on how to throttle / rate limit their mirror server? Mine sits on a gig-e connection, and I just got a call from campus IT questioning the amount of bandwidth I'm using...Right now, I'm running it "fully open", but I may have to restrict that, at least during certain hours. I run http, ftp, and rsync on my server.
Rsync I haven't throttled bandwith usage, but have capped the max number of concurrent connections. Unless someone is creating a fresh copy of the mirror, then rsync is rather efficient. I also do not see a lot of users connecting over rsync.
If you are using vsftp, then you could add something like this to your /etc/vsftp/vsftp.conf:
# Max transfer rate 10 Mb anon_max_rate=10485760
HTTP I haven't looked at that much, but EPEL has mod_bw that restricts the maximum connections and bandwidth. I have not tested mod_bw though, as bandwidth luckily hasn't become an issue yet for our mirror...
-Jonathan
On 03/22/2011 11:24 AM, Jonathan Thurman wrote:
From: centos-mirror-bounces@centos.org [centos-mirror-bounces@centos.org] on behalf of Jim Kusznir
So do users have a suggestion on how to throttle / rate limit their mirror server? Mine sits on a gig-e connection, and I just got a call from campus IT questioning the amount of bandwidth I'm using...Right now, I'm running it "fully open", but I may have to restrict that, at least during certain hours. I run http, ftp, and rsync on my server.
HTTP I haven't looked at that much, but EPEL has mod_bw that restricts the maximum connections and bandwidth. I have not tested mod_bw though, as bandwidth luckily hasn't become an issue yet for our mirror...
We used to cap HTTP and mod_bw worked well for us with exclusions for our local IP ranges in order to provide them maximum speed. We don't cap anymore though and I've never capped FTP or rsync. I didn't open up other protocols until we decided to stop capping.
On 03/22/2011 12:24 PM, Jonathan Thurman wrote:
From: centos-mirror-bounces@centos.org [centos-mirror-bounces@centos.org] on behalf of Jim Kusznir
So do users have a suggestion on how to throttle / rate limit their mirror server? Mine sits on a gig-e connection, and I just got a call from campus IT questioning the amount of bandwidth I'm using...Right now, I'm running it "fully open", but I may have to restrict that, at least during certain hours. I run http, ftp, and rsync on my server.
Rsync I haven't throttled bandwith usage, but have capped the max number of concurrent connections. Unless someone is creating a fresh copy of the mirror, then rsync is rather efficient. I also do not see a lot of users connecting over rsync.
If you are using vsftp, then you could add something like this to your /etc/vsftp/vsftp.conf:
# Max transfer rate 10 Mb anon_max_rate=10485760
...
We're running a ftp-only (so far) mirror that I'm about to announce to the list. It's my understanding that you need to adjust anon_max_rate *and* max_clients or else you could get, for example, 100 connections using a max transfer rate of 10Mb. So far I'm limiting our ftp connections/rate thusly:
anon_max_rate=5000000 max_clients=20
so that we can't get hit by over 100Mb of traffic. Please let me know if this doesn't sound right, or if 20 clients maximum is way too wimpy to be a useful mirror. Also, I'm assuming that since we're talking about network transfer rate, the rate doesn't need to be x(1024^2), just x(1000^2)?
I'm going to set the speed of the interface and the switch to 100, as a stopgap as well.
On 03/22/2011 12:24 PM, Jonathan Thurman wrote:
If you are using vsftp, then you could add something like this to your /etc/vsftp/vsftp.conf:
# Max transfer rate 10 Mb anon_max_rate=10485760
We're running a ftp-only (so far) mirror that I'm about to announce to the list. It's my understanding that you need to adjust anon_max_rate *and* max_clients or else you could get, for example, 100 connections using a max transfer rate of 10Mb. So far I'm limiting our ftp connections/rate thusly:
Good point. All the man page says is "The maximum data transfer rate permitted, in bytes per second, for anonymous clients.", but does not specifically state that is for ALL anonymous clients. The wording for local_max_rate leads me to believe that the rates are per-user, but I haven't reviewed the source.
anon_max_rate=5000000 max_clients=20
You might want to add max_per_ip as well, otherwise a single system could use up all 20 of your client connections. Don't make it too small though, as there are a lot of systems behind NAT.
so that we can't get hit by over 100Mb of traffic. Please let me know if this doesn't sound right, or if 20 clients maximum is way too wimpy to be a useful mirror. Also, I'm assuming that since we're talking about network transfer rate, the rate doesn't need to be x(1024^2), just x(1000^2)?
anon_max_rate is a cap in bytes per second, so it does not have to be a multiple of anything. If you like limiting the number of bytes to a prime number, it should work =)
-Jonathan
For the record, what I ended up deploying was linux network-stack level traffic control with the HTB shaper. I ran into some strangeness with my filter rules, which were preventing my local traffic from getting exempted. I finally removed the default rule from the root shaper and put a u32 catch-all rule in pointing to the class with the restriction in it.
This way it will catch ALL outbound bandwidth, as the local ITS was getting pretty ansy and they didn't care about what modes were in use, just that the absolute total amount of packets leaving my machine was at or below 50mbits/s.
Oh well....
--Jim
On Wed, Mar 23, 2011 at 1:13 PM, Jonathan Thurman JThurman@nwresd.k12.or.us wrote:
On 03/22/2011 12:24 PM, Jonathan Thurman wrote:
If you are using vsftp, then you could add something like this to your /etc/vsftp/vsftp.conf:
# Max transfer rate 10 Mb anon_max_rate=10485760
We're running a ftp-only (so far) mirror that I'm about to announce to the list. It's my understanding that you need to adjust anon_max_rate *and* max_clients or else you could get, for example, 100 connections using a max transfer rate of 10Mb. So far I'm limiting our ftp connections/rate thusly:
Good point. All the man page says is "The maximum data transfer rate permitted, in bytes per second, for anonymous clients.", but does not specifically state that is for ALL anonymous clients. The wording for local_max_rate leads me to believe that the rates are per-user, but I haven't reviewed the source.
anon_max_rate=5000000 max_clients=20
You might want to add max_per_ip as well, otherwise a single system could use up all 20 of your client connections. Don't make it too small though, as there are a lot of systems behind NAT.
so that we can't get hit by over 100Mb of traffic. Please let me know if this doesn't sound right, or if 20 clients maximum is way too wimpy to be a useful mirror. Also, I'm assuming that since we're talking about network transfer rate, the rate doesn't need to be x(1024^2), just x(1000^2)?
anon_max_rate is a cap in bytes per second, so it does not have to be a multiple of anything. If you like limiting the number of bytes to a prime number, it should work =)
-Jonathan _______________________________________________ CentOS-mirror mailing list CentOS-mirror@centos.org http://lists.centos.org/mailman/listinfo/centos-mirror
-
James A. Peltier -
Jim Kusznir -
Jonathan Thurman -
Kevin Stange -
Nick Olsen -
Paul Stewart -
Simeon Berkley