Hi everyone
At http://www.centos.org/modules/newbb/viewtopic.php?topic_id=2217&forum=6 I expressed some dismay at the fact that several of the update mirrors are broken or outdated. I also proposed some remedies for the problem.
Since nobody has replied there, I thought I'd better bring the issue here.
I wrote a script which checks header.info on all parts of both distributions on all update mirrors using http headers only, thus a totally negligible amount of bandwidth, and then modifies sdb_ldap DNS zone records for mirror.centos.org to only reflect good mirrors. Of course it could run against a plain vanilla BIND too, but sdb_ldap has the great advantage that it eliminates the need to reload after every zone change, so that removing bad mirrors from and re-inserting repaired ones into the round-robin can be done really often.
To go along with the script, I also wrote an sdb_ldap .ldif for mirror.centos.org.
Is there any interest to implement any of this?
Z
On Wed, 9 Nov 2005, Zenon Panoussis wrote:
Hi everyone
At http://www.centos.org/modules/newbb/viewtopic.php?topic_id=2217&forum=6 I expressed some dismay at the fact that several of the update mirrors are broken or outdated. I also proposed some remedies for the problem.
I think you may have misunderstood the fact that all of the servers that are referenced as mirror.centos.org are managed and maintained by the CentOS team - they are not external mirrors.
Yes - at certain times those machines may not respond - usually due to excessive load caused by a major (Ux) update - although hardware falure is not unknown.
Since nobody has replied there, I thought I'd better bring the issue here.
I wrote a script which checks header.info on all parts of both distributions on all update mirrors using http headers only, thus a totally negligible amount of bandwidth, and then modifies sdb_ldap DNS zone records for mirror.centos.org to only reflect good mirrors. Of course it could run against a plain vanilla BIND too, but sdb_ldap has the great advantage that it eliminates the need to reload after every zone change, so that removing bad mirrors from and re-inserting repaired ones into the round-robin can be done really often.
Sounds like a good and useful idea - although dynamic dns updates certainly dont require bind to be reloaded ... updates being kept in the .jnl file and the zone file is only rewritten at intervals, to be used on a reload/restart. Having said that - I havent experience of bind ddns being used for rrdns ...
Is the sdb_ldap part of CentOS or sometng that would need to be installed/maintained separately ??
To go along with the script, I also wrote an sdb_ldap .ldif for mirror.centos.org.
Is there any interest to implement any of this?
Yes - definitely
Especially if it may also allow us to extend the mirror. schema out to external mirrors, and tie in with the stuff we are working on for mirrorlist ...
Regards Lance
Lance Davis wrote:
I think you may have misunderstood the fact that all of the servers that are referenced as mirror.centos.org are managed and maintained by the CentOS team - they are not external mirrors.
Indeed, looking at where whey are placed I assumed they were externally managed. Yet, regardless who manages them, their bad responses cause problems.
Yes - at certain times those machines may not respond - usually due to excessive load caused by a major (Ux) update - although hardware falure is not unknown.
I have seen three different problems on numerous ocassions:
- 404 on header.info - "X is not a gzipped file" (probably refering to some header, I can't remember exactly) - .hdr files are present but the corresponding .rpm files are missing, causing yum to crash. This was a most common and widespread problem during the 4 U2 release.
Each of these requires a different remedy. My checker deals with the first two. The last problem could be eliminated by putting .rpm before .hdr in the rsync mirroring scripts.
[mirror checking script]
Sounds like a good and useful idea - although dynamic dns updates certainly dont require bind to be reloaded ... updates being kept in the .jnl file
Uhm, I have a lot of experience with ldap and none at all with rndc, that's why I grabbed for ldap. But of course, rndc could very well be used instead.
Having said that - I havent experience of bind ddns being used for rrdns ...
I have been using BIND with sbd_ldap as DDNS for a few months now and it works just fine, although I've never had the opportunity to test it under really heavy loads. As for the round-robin part, I don't think BIND cares where it gets its zone data from, as long as it gets it.
Is the sdb_ldap part of CentOS or sometng that would need to be installed/maintained separately ??
It's comes with FC4. I completely missed the fact, so when I needed it I created a patched the CentOS 4 srpm instead of just grabbing the FC4 srpm. The patch is at https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=163190
Is there any interest to implement any of this?
Yes - definitely
I that case, get the whole thing at http://www.provocation.net/mirrorchecker.tar.bz2 and take a look at it. I'm sure there is plenty of room for improvement, but all the basics are there. Run the script and you will find that two hosts in mirror.centos.org have a problem right now.
Especially if it may also allow us to extend the mirror. schema out to external mirrors, and tie in with the stuff we are working on for mirrorlist ...
That's one solution, surely better than what's there now. All that's needed is a virtual mirror.centos.org server on each mirror, with its document root pointing at the directory that contains centos. This would solve the problem of different paths to centos on each server.
On the other hand, yum is unable to recover if it meets a bad mirror in a round-robined system. Using several mirrors with different hostnames in sources can allow yum to proceed elsewhere if the particular mirror.centos.org it first goes to is broken.
Z
Hi Zenon,
Zenon Panoussis wrote:
Hi everyone
At http://www.centos.org/modules/newbb/viewtopic.php?topic_id=2217&forum=6 I expressed some dismay at the fact that several of the update mirrors are broken or outdated. I also proposed some remedies for the problem.
Since nobody has replied there, I thought I'd better bring the issue here.
I wrote a script which checks header.info on all parts of both distributions on all update mirrors using http headers only, thus a totally negligible amount of bandwidth, and then modifies sdb_ldap DNS zone records for mirror.centos.org to only reflect good mirrors. Of course it could run against a plain vanilla BIND too, but sdb_ldap has the great advantage that it eliminates the need to reload after every zone change, so that removing bad mirrors from and re-inserting repaired ones into the round-robin can be done really often.
To go along with the script, I also wrote an sdb_ldap .ldif for mirror.centos.org.
Is there any interest to implement any of this?
You might want to keep in mind that mirror.centos.org are machines that are run by us, the centos project, on donated dedicated servers. The mirrors listed on http://www.centos.org/mirrors are _NOT_ a part of mirror.centos.org
However this issue of external mirror state, is an issue that came up a few weeks back and Lance has written a state checking script. We plan on implementing this to generate mirrorlist data that can then in turn be used by yum to make sure users get only 'updated' mirrors.
- K
Karanbir Singh wrote:
Hi
You might want to keep in mind that mirror.centos.org are machines that are run by us, the centos project, on donated dedicated servers. The mirrors listed on http://www.centos.org/mirrors are _NOT_ a part of mirror.centos.org
Yet, two of them are broken as we speak, 83.149.120.9 only partially:
--16:41:30-- http://72.21.40.10/centos-4/4/updates/i386/headers/header.info => `header.info' Connecting to 72.21.40.10:80... connected. HTTP request sent, awaiting response... HTTP/1.1 404 Not Found Date: Wed, 09 Nov 2005 15:41:30 GMT Server: Apache/2.0.46 (CentOS) Keep-Alive: timeout=15, max=1000 Connection: Keep-Alive Content-Type: text/html; charset=iso-8859-1 16:41:31 ERROR 404: Not Found.
--16:41:30-- http://83.149.120.9/centos-3/3/updates/i386/headers/header.info => `header.info' Connecting to 83.149.120.9:80... connected. HTTP request sent, awaiting response... HTTP/1.1 404 Not Found Date: Wed, 09 Nov 2005 15:41:32 GMT Server: Apache/2.0.52 (CentOS) Keep-Alive: timeout=15, max=1000 Connection: Keep-Alive Content-Type: text/html; charset=iso-8859-1 16:41:33 ERROR 404: Not Found.
Z
Zenon Panoussis wrote:
Karanbir Singh wrote:
Hi
You might want to keep in mind that mirror.centos.org are machines that are run by us, the centos project, on donated dedicated servers. The mirrors listed on http://www.centos.org/mirrors are _NOT_ a part of mirror.centos.org
Yet, two of them are broken as we speak, 83.149.120.9 only partially:
--16:41:30-- http://72.21.40.10/centos-4/4/updates/i386/headers/header.info => `header.info' Connecting to 72.21.40.10:80... connected. HTTP request sent, awaiting response... HTTP/1.1 404 Not Found Date: Wed, 09 Nov 2005 15:41:30 GMT Server: Apache/2.0.46 (CentOS) Keep-Alive: timeout=15, max=1000 Connection: Keep-Alive Content-Type: text/html; charset=iso-8859-1 16:41:31 ERROR 404: Not Found.
--16:41:30-- http://83.149.120.9/centos-3/3/updates/i386/headers/header.info => `header.info' Connecting to 83.149.120.9:80... connected. HTTP request sent, awaiting response... HTTP/1.1 404 Not Found Date: Wed, 09 Nov 2005 15:41:32 GMT Server: Apache/2.0.52 (CentOS) Keep-Alive: timeout=15, max=1000 Connection: Keep-Alive Content-Type: text/html; charset=iso-8859-1 16:41:33 ERROR 404: Not Found.
you should really be looking at /centos/4/ and /centos/3/ and not /centos-4/ and /centos-3/ ( ref: to the way yum repo's are setup presently ).
However, the issues you pointed out are valid - and need fixing.
- K
Karanbir Singh wrote:
Yet, two of them are broken as we speak, 83.149.120.9 only partially:
http://72.21.40.10/centos-4/4/updates/i386/headers/header.info => `header.info' HTTP/1.1 404 Not Found
you should really be looking at /centos/4/ and /centos/3/ and not /centos-4/ and /centos-3/ ( ref: to the way yum repo's are setup presently ).
You're right.
When I look in the right place, I still get a 404 though:
# host mirror.centos.org mirror.centos.org has address 66.147.238.146 mirror.centos.org has address 72.21.40.10 mirror.centos.org has address 83.149.120.9 mirror.centos.org has address 131.211.85.22 mirror.centos.org has address 131.211.85.43 mirror.centos.org has address 205.209.141.148 mirror.centos.org has address 206.51.236.66 mirror.centos.org has address 64.34.175.102 mirror.centos.org has address 64.38.239.200 mirror.centos.org has address 64.92.174.74
--17:02:42-- http://72.21.40.10/centos/4/os/i386/headers/header.info => `header.info' Connecting to 72.21.40.10:80... connected. HTTP request sent, awaiting response... HTTP/1.1 404 Not Found Date: Wed, 09 Nov 2005 16:02:41 GMT Server: Apache/2.0.46 (CentOS) Keep-Alive: timeout=15, max=1000 Connection: Keep-Alive Content-Type: text/html; charset=iso-8859-1 17:02:42 ERROR 404: Not Found.
Plain http://72.21.40.10/ redirects to lists.centos.org . Am I missing something again or should this machine not be part of the round-robin?
Z
On Wed, 2005-11-09 at 17:12 +0100, Zenon Panoussis wrote:
Using the IP address of the machine in question won't work ... as mirror.centos.org (or in the case of the machine you are testing, centosg) are virtual hosts on that machine.
that file is there and just fine if you use the FQDN for that server (centosg.centos.org) OR if mirror.centos.org picks that machine to resolve to.
The file not found issue that we were seeing before was simply because we could not start more than 256 httpd process on our servers.
-
Johnny Hughes -
Karanbir Singh -
Lance Davis -
Zenon Panoussis