Hi Everyone, I'm new here. I just caught this thread and checked my yum.conf. It does indeed point to the main mirror. May I suggest that the default yum.conf file contain a links to some of the public mirrors, or none at all instead of the main mirror.
That way, it doesn't point to the main mirror, which would be reserved for public mirrors. Then in the Docs show an example yum.conf that shows what is acceptable.
Another suggestion would be to put a few public mirrors in the yum.conf file and comment them out. Then put in a comment saying "Here are a few public mirrors, but you can get mirrors closer to you by looking at http://www.centos.org/download/mirrors"
Matt Shields
On Sat, 6 Nov 2004 15:34:53 -0800 (PST), Rick Graves gravesricharde@yahoo.com wrote:
Lance,
Let's go at this from another angle.
I am absolutely certain having analysed the logs that the excess bandwidth was caused by people downloading .iso images for centos from our master mirror(s) ....
How could this happen?
Here is my explanation:
- As far as I am aware, the cAos site does NOT ask
distro users to download from public mirrors, rather than from the master mirror.
On install, yum.conf points to the master mirror.
Even if a distro user puts in a custom yum.conf
file that points to a public mirror, "yum update" can remove the custom yum.conf file, and put in its place a yum.conf that points to the master mirror.
- From 2 & 3, I got the impression that it is OK,
even encouraged, to download directly from the master mirror. This is supported by the lack of any request not to do so, 1.
- Maybe lots of other distro users got the same
impression as 4.
That is my explanation.
Does anyone have a better explanation for Lance's analysis of the logs?
Rick _______________________________________________ CentOS mailing list CentOS@caosity.org http://lists.caosity.org/mailman/listinfo/centos
On Sun, 7 Nov 2004, Matt Shields wrote:
I'm new here. I just caught this thread and checked my yum.conf. It does indeed point to the main mirror. May I suggest that the default yum.conf file contain a links to some of the public mirrors, or none at all instead of the main mirror.
That way, it doesn't point to the main mirror, which would be reserved for public mirrors. Then in the Docs show an example yum.conf that shows what is acceptable.
Another suggestion would be to put a few public mirrors in the yum.conf file and comment them out. Then put in a comment saying "Here are a few public mirrors, but you can get mirrors closer to you by looking at http://www.centos.org/download/mirrors"
The clamav projects recently started with different mirrors with an ISO country code. They have db.CC.clamav.net and when installing a package the configuration is being changed based on the configuration in /etc/sysconfig/clock.
The code can be found here (see %post section):
http://svn.rpmforge.net/svn/trunk/rpms/clamav/clamav.spec
And it would allow you to maintain and balance load based on the DNS zone information.
Although of course the number of network hops and network latency is not directly related to the physical distance (or even country borders).
-- dag wieers, dag@wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors]
Since a way to automatically detect which is the closest mirror may take a while, and there are a ton of people still pointed at the master mirror server I decided to do a test.
I got the Yum source and recompiled it using a new yum.conf and upped the version number. Then put it in my local repository(I cache all packages locally so I only download them once) and then I ran yum update and it saw the new package and updated yum.conf. Although it wouldn't be good to assume that people wanted to use 1 specific mirror, it would get the burden off the master mirror.
You could also push a broken yum.conf where the baseurl is something like http://go.to.caosity.org.to.change.your.mirror/ then when someone does a yum update they will get an error "Error getting file http://go.to.caosity.org.to.change.your.mirror/" and post a notice on the main webpage telling people the reason it was broken and give them the link to the mirrors page and the explain the reason why yum was intentionally broken.
Another option would be to setup a list of whitelisted IPs (active mirrors) that can use the master mirror server, all other addresses are blacklisted. This wouldn't involve the end user updating yum, and would protect your bandwidth.
Although none of these are the ideal situation, you need to do something to protect yourselves from another high isp bill. These would be quick fixes till permanent solution can be found.
Matt
On Mon, 8 Nov 2004 00:37:41 +0100 (CET), Dag Wieers dag@wieers.com wrote:
On Sun, 7 Nov 2004, Matt Shields wrote:
I'm new here. I just caught this thread and checked my yum.conf. It does indeed point to the main mirror. May I suggest that the default yum.conf file contain a links to some of the public mirrors, or none at all instead of the main mirror.
That way, it doesn't point to the main mirror, which would be reserved for public mirrors. Then in the Docs show an example yum.conf that shows what is acceptable.
Another suggestion would be to put a few public mirrors in the yum.conf file and comment them out. Then put in a comment saying "Here are a few public mirrors, but you can get mirrors closer to you by looking at http://www.centos.org/download/mirrors"
The clamav projects recently started with different mirrors with an ISO country code. They have db.CC.clamav.net and when installing a package the configuration is being changed based on the configuration in /etc/sysconfig/clock.
The code can be found here (see %post section):
http://svn.rpmforge.net/svn/trunk/rpms/clamav/clamav.specAnd it would allow you to maintain and balance load based on the DNS zone information.
Although of course the number of network hops and network latency is not directly related to the physical distance (or even country borders).
-- dag wieers, dag@wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors]
You should re-write yum so that it searches for a different [random] mirror for each rpm it needs.
That way the load is spread.
I could run a mirror for you, I would be willing to donate 1/4 meg uncontended when my 2 meg feed goes in in december....
I don't have much bandwidth spare but I want to do my bit...
P.
Matt Shields wrote:
Since a way to automatically detect which is the closest mirror may take a while, and there are a ton of people still pointed at the master mirror server I decided to do a test.
I got the Yum source and recompiled it using a new yum.conf and upped the version number. Then put it in my local repository(I cache all packages locally so I only download them once) and then I ran yum update and it saw the new package and updated yum.conf. Although it wouldn't be good to assume that people wanted to use 1 specific mirror, it would get the burden off the master mirror.
You could also push a broken yum.conf where the baseurl is something like http://go.to.caosity.org.to.change.your.mirror/ then when someone does a yum update they will get an error "Error getting file http://go.to.caosity.org.to.change.your.mirror/" and post a notice on the main webpage telling people the reason it was broken and give them the link to the mirrors page and the explain the reason why yum was intentionally broken.
Another option would be to setup a list of whitelisted IPs (active mirrors) that can use the master mirror server, all other addresses are blacklisted. This wouldn't involve the end user updating yum, and would protect your bandwidth.
Although none of these are the ideal situation, you need to do something to protect yourselves from another high isp bill. These would be quick fixes till permanent solution can be found.
Matt
On Mon, 8 Nov 2004 00:37:41 +0100 (CET), Dag Wieers dag@wieers.com wrote:
On Sun, 7 Nov 2004, Matt Shields wrote:
I'm new here. I just caught this thread and checked my yum.conf. It does indeed point to the main mirror. May I suggest that the default yum.conf file contain a links to some of the public mirrors, or none at all instead of the main mirror.
That way, it doesn't point to the main mirror, which would be reserved for public mirrors. Then in the Docs show an example yum.conf that shows what is acceptable.
Another suggestion would be to put a few public mirrors in the yum.conf file and comment them out. Then put in a comment saying "Here are a few public mirrors, but you can get mirrors closer to you by looking at http://www.centos.org/download/mirrors"
The clamav projects recently started with different mirrors with an ISO country code. They have db.CC.clamav.net and when installing a package the configuration is being changed based on the configuration in /etc/sysconfig/clock.
The code can be found here (see %post section):
http://svn.rpmforge.net/svn/trunk/rpms/clamav/clamav.specAnd it would allow you to maintain and balance load based on the DNS zone information.
Although of course the number of network hops and network latency is not directly related to the physical distance (or even country borders).
-- dag wieers, dag@wieers.com, http://dag.wieers.com/ -- [Any errors in spelling, tact or fact are transmission errors]
CentOS mailing list CentOS@caosity.org http://lists.caosity.org/mailman/listinfo/centos
On Tue, 2004-11-16 at 13:52, Peter Farrow wrote:
You should re-write yum so that it searches for a different [random] mirror for each rpm it needs.
That way the load is spread.
I could run a mirror for you, I would be willing to donate 1/4 meg uncontended when my 2 meg feed goes in in december....
I don't have much bandwidth spare but I want to do my bit...
That would suck for the user, quite a lot.
The whole point of going until you find a reliable mirror is to make it better for the user.
the best way to offload the load is to organize the mirrors. Have the primary be mirrors-only and everyone else connect to a list of mirrors.
-sv
I thought about that about 20 secs after I sent the mail, thinking about partly retrieved updates & missing rpms,
so yes I agree....
P.
seth vidal wrote:
On Tue, 2004-11-16 at 13:52, Peter Farrow wrote:
You should re-write yum so that it searches for a different [random] mirror for each rpm it needs.
That way the load is spread.
I could run a mirror for you, I would be willing to donate 1/4 meg uncontended when my 2 meg feed goes in in december....
I don't have much bandwidth spare but I want to do my bit...
That would suck for the user, quite a lot.
The whole point of going until you find a reliable mirror is to make it better for the user.
the best way to offload the load is to organize the mirrors. Have the primary be mirrors-only and everyone else connect to a list of mirrors.
-sv
seth vidal wrote:
On Tue, 2004-11-16 at 13:52, Peter Farrow wrote:
You should re-write yum so that it searches for a different [random] mirror for each rpm it needs.
That way the load is spread.
I could run a mirror for you, I would be willing to donate 1/4 meg uncontended when my 2 meg feed goes in in december....
I don't have much bandwidth spare but I want to do my bit...
That would suck for the user, quite a lot.
The whole point of going until you find a reliable mirror is to make it better for the user.
the best way to offload the load is to organize the mirrors. Have the primary be mirrors-only and everyone else connect to a list of mirrors.
It would be nice if you could do some sort of "bittorrent" like download. This way bandwidth could be more easily balanced between all mirrors.
Another thought would be to encourage the use some sort of distributed caching, something like http://www.ircache.net/.
-Dave
-
Dag Wieers -
David Rees -
Matt Shields -
Peter Farrow -
seth vidal