hi, it seems that the anaconda src.rpm missing from the mirror sites SRPMS dir. is there any reason for this or it´s just a bug? thanks in advance.
On Wed, Apr 8, 2009 at 9:52 PM, Farkas Levente lfarkas@lfarkas.org wrote:
it seems that the anaconda src.rpm missing from the mirror sites SRPMS dir. is there any reason for this or it´s just a bug?
I don't want to offend you, but it is appreciated that you first search the archives to see if your question is already answered, or any other support mechanism. Which is the fact in this case.
So, see the archives for your answer.
Kind regards, Tim
On Wednesday 08 April 2009, Farkas Levente wrote:
hi, it seems that the anaconda src.rpm missing from the mirror sites SRPMS dir. is there any reason for this or it´s just a bug? thanks in advance.
Just syncing slowly is what I've heard.
/Peter
On Wed, 8 Apr 2009, Peter Kjellstrom wrote:
On Wednesday 08 April 2009, Farkas Levente wrote:
hi, it seems that the anaconda src.rpm missing from the mirror sites SRPMS dir. is there any reason for this or it�s just a bug? thanks in advance.
Just syncing slowly is what I've heard.
I hope that consideration will be given in future releases to sync the SRPMS before the binaries, do avoid this time skew.
--- Charlie
On Wed, 8 Apr 2009, Charlie Brady wrote:
On Wed, 8 Apr 2009, Peter Kjellstrom wrote:
On Wednesday 08 April 2009, Farkas Levente wrote:
hi, it seems that the anaconda src.rpm missing from the mirror sites SRPMS dir. is there any reason for this or it?s just a bug? thanks in advance.
Just syncing slowly is what I've heard.
I hope that consideration will be given in future releases to sync the SRPMS before the binaries, do avoid this time skew.
And then we get thousands of people asking why the SRPMs are available but not the RPMs and the ISOs. I prefer this :-)
On Wed, 8 Apr 2009, Charlie Brady wrote:
Just syncing slowly is what I've heard.
I hope that consideration will be given in future releases to sync the SRPMS before the binaries, do avoid this time skew.
Let's see -- Perhaps one in 200 people USING CentOS binaries, use CentOS SRPMs. Next look at the relentless railing and carping about 'delays' and 'lateness'. Then look at the negligible (or at least minimal) responses to repeated requests for donation of additional resources to the project.
SRPMs first is not likely a course likely to serve the largest number first, nor cut the pain of having to listen to the the larger source of thankless, thoughtless whimpering, I'd say
A person wanting earlier SRPM access probably has to put up resources to facilitate such a path (I am aware of Shad Lord's recent mirror offer). More bandwidth at the needed points [and avoiding fiber cuts] is harder to donate, sadly.
Mirror flap seems to still be a problem as well, probably due to some mirrors not syncing against sub-masters properly, but rather trying to cross-sync, and so confusing inferior unofficial sub-sub-mirrors that have beenm inprovidently editted in (based on main IRC channel diagnosis)
I would be thrilled to have a simultaneous coordinated release, but the 'leak' of 'patched' torrent instances, and at least two mirrors opening the full ISO set before the coordinated bit flip date and time, leave rather a bad outlook to me as to the ability to make things better through such an 'inverted as to demand' approach
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
-- Russ herrold
R P Herrold wrote:
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
Do we know exactly what the problem is? That is, why does yum do something different after a 'yum clean all'? Shouldn't it be trying all the mirrors anyway if it fails to get a file for any reason? Otherwise, what's the point of having the list that generally screws up caching?
I don't think the answer is to expect the repos to be perfect but rather to make the clients recover without intervention (and without killing the good repos too...).
On Wed, 8 Apr 2009, Les Mikesell wrote:
R P Herrold wrote:
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
Do we know exactly what the problem is?
I listed some of them in what you trimmed, Les, -- mirror bitflip frontrunning (inadvertent in one case we ran down), iso leak (collateral damage from the first), some in the thundering herd who should know better raising expectations started a cascade outbreak of 'Latest and Greatest' disease; no good deed goes unpunished, it seems
That is, why does yum do something different after a 'yum clean all'? Shouldn't it be trying all the mirrors anyway if it fails to get a file for any reason? Otherwise, what's the point of having the list that generally screws up caching?
I do not see that yum failover is **not** working; indeed it seems to be working just fine in my testing against a 'as designed' "centos-release" package as we ship it. The outcome _I_ see when I hit 'centos.mirror.nac.net' is the failure, a failover, and a success on a later listed peer. The 127.0.0.2 workaround will permit you to test this as well (simulating a dead mirrorlist entry); transparent proxies are out of our control by definition.
I am not so interested in trying slow motion debug via mailing list of what a person's setup is, and will (and do) read bugs.centos.org for reports from people who file a formal report
I don't think the answer is to expect the repos to be perfect but rather to make the clients recover without intervention (and without killing the good repos too...).
patches to yum upstream are still welcome, I assume. Feel free to ask Seth, et al.
-- Russ herrold
R P Herrold wrote:
That is, why does yum do something different after a 'yum clean all'? Shouldn't it be trying all the mirrors anyway if it fails to get a file for any reason? Otherwise, what's the point of having the list that generally screws up caching?
I do not see that yum failover is **not** working; indeed it seems to be working just fine in my testing against a 'as designed' "centos-release" package as we ship it. The outcome _I_ see when I hit 'centos.mirror.nac.net' is the failure, a failover, and a success on a later listed peer. The 127.0.0.2 workaround will permit you to test this as well (simulating a dead mirrorlist entry); transparent proxies are out of our control by definition.
Dead mirrors are different from live mirrors that don't have a file. I can't simulate the latter, which seemed to be the source of the problem.
Most of my x86 updates required this:
yum -y --exclude tk --exclude linuxwacom --exclude wxGTK --exclude nss-devel --exclude 'rpm-*' --exclude popt --exclude kernel* update
on the first pass or they failed with one or more missing packages.
After this completed and I did a 'yum clean all' a subsequent 'yum update' picked up all of the rest, including the previously missing package(s). Proxies were involved in most cases, but not transparent proxies.
I am not so interested in trying slow motion debug via mailing list of what a person's setup is, and will (and do) read bugs.centos.org for reports from people who file a formal report
I don't know how to submit a sensible bug report about something that is not repeatable. I have an odd mix of 3rd party repositories enabled but that shouldn't cause a 'missing' package in the first place and didn't change between runs. It also seemed to be the same thing happening to others, since I got most of the --exclude's from the mail list postings. The only thing visibly different between runs was that after the "yum clean all" it went through the motions of recomputing the fastest mirror.
I don't think the answer is to expect the repos to be perfect but rather to make the clients recover without intervention (and without killing the good repos too...).
patches to yum upstream are still welcome, I assume. Feel free to ask Seth, et al.
Seth doesn't like my ranting any more than you do. Usually it is about the mirrorlist rotation screwing up caching - which isn't even his fault, but this time the download speeds were fast enough that I didn't care about the duplication. So your mirrors have great performance even if some contents are missing.
On Wed, 2009-04-08 at 16:50 -0500, Les Mikesell wrote:
That is, why does yum do something different after a 'yum clean all'? Shouldn't it be trying all the mirrors anyway if it fails to get a file for any reason?
Because "yum clean all" will delete your mirrorlist.txt file, so then you could be (and I assume are) talking to a different set of mirrors on any subsequent runs. "yum clean expire-cache" is the less expensive way of doing the same thing.
James Antill wrote:
On Wed, 2009-04-08 at 16:50 -0500, Les Mikesell wrote:
That is, why does yum do something different after a 'yum clean all'? Shouldn't it be trying all the mirrors anyway if it fails to get a file for any reason?
Because "yum clean all" will delete your mirrorlist.txt file, so then you could be (and I assume are) talking to a different set of mirrors on any subsequent runs.
OK, but these runs were a few minutes apart - and days after the problems started. Why would the mirrorlists be different other than perhaps having a different sort order? I thought the point of the lists was to have them all for retry/failover.
"yum clean expire-cache" is the less expensive way of doing the same thing.
That's not exactly intuitive - nor is why it should help. Why can't yum do this on its own if the repo metadata says you need a file but the repo says the file isn't there? It is also kind of painful recomputing the fastest mirror and doing failovers that hit ftp:// urls in my locations where the proxies don't handle ftp. Is there a way to bypass them that won't kill the (obviously needed) ability to find freshly listed repos?
R P Herrold wrote:
I would be thrilled to have a simultaneous coordinated release, but the 'leak' of 'patched' torrent instances, and at least two mirrors opening the full ISO set before the coordinated bit flip date and time, leave rather a bad outlook to me as to the ability to make things better through such an 'inverted as to demand' approach
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
Would it help any to keep a file in the tree that is *supposed* to be inaccessible to end-users via HTTP/FTP/Rsync? The mirrormon scripts (or some other regular check) could test to see if the file was accessible, and if so, warn the admin, and perhaps take the host out of rotation.
This might help catch issues like those that allowed files to leak out pre-bitflip.
-Brandon
on 4-8-2009 2:36 PM R P Herrold spake the following:
On Wed, 8 Apr 2009, Charlie Brady wrote:
Just syncing slowly is what I've heard.
I hope that consideration will be given in future releases to sync the SRPMS before the binaries, do avoid this time skew.
Let's see -- Perhaps one in 200 people USING CentOS binaries, use CentOS SRPMs. Next look at the relentless railing and carping about 'delays' and 'lateness'. Then look at the negligible (or at least minimal) responses to repeated requests for donation of additional resources to the project.
SRPMs first is not likely a course likely to serve the largest number first, nor cut the pain of having to listen to the the larger source of thankless, thoughtless whimpering, I'd say
A person wanting earlier SRPM access probably has to put up resources to facilitate such a path (I am aware of Shad Lord's recent mirror offer). More bandwidth at the needed points [and avoiding fiber cuts] is harder to donate, sadly.
Mirror flap seems to still be a problem as well, probably due to some mirrors not syncing against sub-masters properly, but rather trying to cross-sync, and so confusing inferior unofficial sub-sub-mirrors that have been improvidently edited in (based on main IRC channel diagnosis)
I would be thrilled to have a simultaneous coordinated release, but the 'leak' of 'patched' torrent instances, and at least two mirrors opening the full ISO set before the coordinated bit flip date and time, leave rather a bad outlook to me as to the ability to make things better through such an 'inverted as to demand' approach
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
-- Russ herrold
The first sync could be iso's with a strategic part cut off. A later rsync could fill this in fairly quickly. Release rpms next for sync. After mirrors are near on line create or at least restore metadata files to sync. If torrent files are released, make sure they are created against the full ISO's before chopping off a chunk. That way they will sit at near completion until the ISO's are actually complete. Peers will have most of the ISO, but it will be unusable until finished, and they can just wait also.
I actually waited until I saw the release announcement before I started a torrent download, and have only yum updated a test server. I am not in a real big hurry, as I already have plenty to do.
"The floggings will continue until morale improves" ;-P
On Apr 8, 2009, at 7:10 PM, Scott Silva wrote:
The first sync could be iso's with a strategic part cut off. A later rsync could fill this in fairly quickly. Release rpms next for sync. After mirrors are near on line create or at least restore metadata files to sync. If torrent files are released, make sure they are created against the full ISO's before chopping off a chunk. That way they will sit at near completion until the ISO's are actually complete. Peers will have most of the ISO, but it will be unusable until finished, and they can just wait also.
Spreading deliberately damaged iso's won't stop the torrents and is a wonderful way to earn a bad reputation.
Otherwise would work just fine, strategically damaged distribution media was even considered @redhat.com before someone realized the flaw.
I actually waited until I saw the release announcement before I started a torrent download, and have only yum updated a test server. I am not in a real big hurry, as I already have plenty to do.
"The floggings will continue until morale improves" ;-P
I'm in favor of the torquemada! Nothing like a dim dark place to reflect upon insanity.
73 de Jeff
on 4-8-2009 4:14 PM Jeff Johnson spake the following:
On Apr 8, 2009, at 7:10 PM, Scott Silva wrote:
The first sync could be iso's with a strategic part cut off. A later rsync could fill this in fairly quickly. Release rpms next for sync. After mirrors are near on line create or at least restore metadata files to sync. If torrent files are released, make sure they are created against the full ISO's before chopping off a chunk. That way they will sit at near completion until the ISO's are actually complete. Peers will have most of the ISO, but it will be unusable until finished, and they can just wait also.
Spreading deliberately damaged iso's won't stop the torrents and is a wonderful way to earn a bad reputation.
But if an iso's .torrent file is created based on the full cd, wouldn't the torrent clients just sit until the iso is actually complete? Maybe someway of witholding one or 2 chunks in the torrent. There would not be any seeds until the ISO's actually were complete.
Otherwise would work just fine, strategically damaged distribution media was even considered @redhat.com before someone realized the flaw.
I actually waited until I saw the release announcement before I started a torrent download, and have only yum updated a test server. I am not in a real big hurry, as I already have plenty to do.
"The floggings will continue until morale improves" ;-P
I'm in favor of the torquemada! Nothing like a dim dark place to reflect upon insanity.
Too bad they never got a remote lart working!
On Apr 8, 2009, at 7:24 PM, Scott Silva wrote:
Spreading deliberately damaged iso's won't stop the torrents and is a wonderful way to earn a bad reputation.
But if an iso's .torrent file is created based on the full cd, wouldn't the torrent clients just sit until the iso is actually complete? Maybe someway of witholding one or 2 chunks in the torrent. There would not be any seeds until the ISO's actually were complete.
Technologically feasible, but practically the same effect on luser reactions imho. Easier to blame the "network" than doing anything else.
(aside) You know 40 years ago they wanted a bigger faster Panama Canal. The proposal was to plant and detonate nukes across Guatemala for sea level access between the Atlantic and the Pacific.
That plan is "technologically feasible", but boy would the Guatemalaans be pissed. ;-)
73 de Jeff
On Wed, Apr 8, 2009 at 7:29 PM, Jeff Johnson n3npq@mac.com wrote:
(aside) You know 40 years ago they wanted a bigger faster Panama Canal. The proposal was to plant and detonate nukes across Guatemala for sea level access between the Atlantic and the Pacific That plan is "technologically feasible", but boy would the Guatemalaans be pissed. ;-)
Perhaps it's time to look at this plan again. With today's weaponry, there wouldn't be any Guatemalaans left to complain. :-P
It could be considered as part of the stimulus package, as it would certainly be a boon to the shipping industry.
On Wednesday 08 April 2009 17:36:18 R P Herrold wrote:
I would be thrilled to have a simultaneous coordinated release, but the 'leak' of 'patched' torrent instances, and at least two mirrors opening the full ISO set before the coordinated bit flip date and time, leave rather a bad outlook to me as to the ability to make things better through such an 'inverted as to demand' approach
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
Russ, Karnbir, et al:
First, a marvelous job on getting the bits out at all; it ain't easy doing this for free (did it on a smaller scale with the PostgreSQL RPMs years ago). I greatly appreciate what the CentOS team does, and how rapidly it gets done. It certainly gets done faster than if I were doing it from the upstream EL's SRPMS.
Now, as to the technical issues, it seems to me that a fully ACID compliant transactional repository mirror system is possibly one way to eliminate most of these issues. Such a system to my knowledge does not yet exist; but, to use an SQL example, something like: BEGIN; MIRROR repo WHERE release = "5.3" AND arch = "x86_64"; COMMIT; and the COMMIT would atomically bring the repo and its mirrors into a consistent state, with already connected clients isolated from the changes as they are being made and with a durability of the result. (yes, wording is intentional). Errors would block the COMMIT until the errors are resolved on a mirror-by-mirror basis; that is, either a mirror shows the full consistent set or it doesn't show anything until it gets the full consistent set and a replica commit occurs. A critical mass of replicas committed would be required to cause a repo commit to avoid overload of individual mirrors.
This is doable now with databases of many gigabytes (I'm in the process of beginning reception of a long term sneakernet push replica/mirror of over 10TB of image data, and the mirror has to be atomic, and it is of course on a database system). But the current pull updating structure doesn't lend itself readily to this.
Incidentally, the MIRROR statement above is intended to project a PUSH arrangement instead of a PULL arrangement; the SQL above would be run on the master to push out the mirrors which could then propagate down hierarchically; reminds me of master-slave database replication with submasters. The fact that some mirrors are partials complicates things even further, though.
Yes, such a system would be a large technical hurdle, and perhaps it would be too complex to work in a loose volunteer arrangement. But surely other upstream projects and distributions have similar issues and needs; perhaps a transactional mirrornet 'system' would be a fine project for someone to start. If one doesn't already exist, that is.
A revision control system can be pressed/abused into this sort of service; monotone, for instance, is/was used by the OpenZaurus people to consistently push out packages, and git is being used by vyatta for similar things, mostly on the developer's side of things (see http://www.vyatta.org/downloads/glendalebuild ). Git has the distributed aspects going for it, but it's not optimal.
Just some ideas.
On Thu, 2009-04-09 at 10:21 -0400, Lamar Owen wrote:
On Wednesday 08 April 2009 17:36:18 R P Herrold wrote:
I would be thrilled to have a simultaneous coordinated release, but the 'leak' of 'patched' torrent instances, and at least two mirrors opening the full ISO set before the coordinated bit flip date and time, leave rather a bad outlook to me as to the ability to make things better through such an 'inverted as to demand' approach
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
Russ, Karnbir, et al:
[...]
Now, as to the technical issues, it seems to me that a fully ACID compliant transactional repository mirror system is possibly one way to eliminate most of these issues. Such a system to my knowledge does not yet exist;
If you want to have a technical talk, you probably want to speak to Matt Domsch who does the mirroring for Fedora (releases 2x a year and rawhide :).
As of "recent" versions of yum, creating repos. with --unique-md-filenames means an atomic move from a fully working old DB to a new one can¹ be done with rename("new-repomd.xml", "repomd.xml"). Which, excepting ext4 ;), is pretty atomic. However my understanding, from speaking to Matt, is that "release day" problems are almost entirely due to getting that very last change to all the mirrors in a timely manner.
Also metalink data can be used to automatically filter out mirrors with old repomd.xml.
However those both require some sacrifices for people using the GA version of yum in CentOS-5, so I'm doubtful that's it's worth the pain.
[...]
But the current pull updating structure doesn't lend itself readily to this.
Trying not to speak too much for Matt, but my understanding is that almost no external mirrors will accept pull mirroring.
¹ My understanding is that no mirrors both with this level of atomic update, but it's possible.
Lamar Owen napsal(a):
Yes, such a system would be a large technical hurdle, and perhaps it would be too complex to work in a loose volunteer arrangement. But surely other upstream projects and distributions have similar issues and needs; perhaps a transactional mirrornet 'system' would be a fine project for someone to start. If one doesn't already exist, that is.
Lamar, there's no need to reinvent the wheel. All interested in how to mirror or how to create CDN look at the MirrorBrain [1] [2]. David Hrbáč
[1] http://mirrorbrain.org/ [2] http://en.opensuse.org/Build_Service/Redirector
R P Herrold wrote:
On Wed, 8 Apr 2009, Charlie Brady wrote:
Just syncing slowly is what I've heard.
I hope that consideration will be given in future releases to sync the SRPMS before the binaries, do avoid this time skew.
Let's see -- Perhaps one in 200 people USING CentOS binaries, use CentOS SRPMs. Next look at the relentless railing and carping about 'delays' and 'lateness'. Then look at the negligible (or at least minimal) responses to repeated requests for donation of additional resources to the project.
SRPMs first is not likely a course likely to serve the largest number first, nor cut the pain of having to listen to the the larger source of thankless, thoughtless whimpering, I'd say
A person wanting earlier SRPM access probably has to put up resources to facilitate such a path (I am aware of Shad Lord's recent mirror offer). More bandwidth at the needed points [and avoiding fiber cuts] is harder to donate, sadly.
Mirror flap seems to still be a problem as well, probably due to some mirrors not syncing against sub-masters properly, but rather trying to cross-sync, and so confusing inferior unofficial sub-sub-mirrors that have beenm inprovidently editted in (based on main IRC channel diagnosis)
I would be thrilled to have a simultaneous coordinated release, but the 'leak' of 'patched' torrent instances, and at least two mirrors opening the full ISO set before the coordinated bit flip date and time, leave rather a bad outlook to me as to the ability to make things better through such an 'inverted as to demand' approach
My $0.02 .... I'd love to be shown a path to avoid the problems on the 5.3 roll-out
it seems i was not clear enough. i ask it because almost all src.rpm are on the mirror site just anaconda missing so for me it seemed it was forgotten. that's why i wrote the original mail. anyway now it's also uploaded:-)
-
Brandon Davidson -
Charlie Brady -
Dag Wieers -
David Hrbáč -
Farkas Levente -
James Antill -
James Pearson -
Jeff Johnson -
Jim Perrin -
Lamar Owen -
Les Mikesell -
Peter Kjellstrom -
R P Herrold -
Ralph Angenendt -
Scott Silva -
Tim Verhoeven