Dear centos-devel'ers,
We have a problem that needs fixing.
The strategy to release testable rpms to dev.centos.org, then request feedback and move the rpms to stable, when we have enough positive - isnt working. We're not getting that feedback, either positive or negative. End result is that we have packages going stale and now some of those packages need updates in themselves. Looking at the logs, I can assure you that people are using those packages, just not telling us anything about them.
There is one chain of though, that such request based qa process's dont work and have been tried by other Open Source projects before and not really had much of a success rate.
Now, there are a few options and alternative process's that I've been thinking about, but rather than throw that out here, I'd really like to see what everyone on this list thinks and what options / process's / workaround's we might be able to come up with here.
So the issue :
The CentOS Project provides packages in addition to the packages from RH's sources. We need to ensure that these packages are tested in a user environment for a while, and only released to the public via the mirror.centos.org network once there is some reasonable surety as to the functional and bugs situation with the package.
ideas / comments / noise welcome..
Hi Karanbir,
The CentOS Project provides packages in addition to the packages from RH's sources. We need to ensure that these packages are tested in a user environment for a while, and only released to the public via the mirror.centos.org network once there is some reasonable surety as to the functional and bugs situation with the package.
I have a question if I may: what should the packages be tested for?
1. Readiness of the packaged software for enterprise environments. 2. Quality of the packaging.
If both, what has most weight? If [1] is important, I suppose the quality can partly be tested with regression tests (if the software includes regression tests, IMO all enterprise-class software should). [2] specifically requires people with enough knowledge of packaging to go over it.
-- Daniel
On Fri, 2006-10-13 at 10:01 -0400, danieldk@pobox.com wrote:
Hi Karanbir,
The CentOS Project provides packages in addition to the packages from RH's sources. We need to ensure that these packages are tested in a user environment for a while, and only released to the public via the mirror.centos.org network once there is some reasonable surety as to the functional and bugs situation with the package.
I have a question if I may: what should the packages be tested for?
- Readiness of the packaged software for enterprise environments.
- Quality of the packaging.
If both, what has most weight? If [1] is important, I suppose the quality can partly be tested with regression tests (if the software includes regression tests, IMO all enterprise-class software should). [2] specifically requires people with enough knowledge of packaging to go over it.
Daniel,
The stuff in the testing repo has already been initially tested (and packaged) by one of the CentOS Developers. It doesn't get in there if it is not at least WorksForMe quality for someone who should understand enterprise ready.
Now I not suggesting that it is perfect, thus the need for testing.
What we need is people to use the products, make sure they work as expected, and tell us it does or does not work for them.
We certainly are also open for suggestions and/or packaging comments and better ways to do things.
I would settle for ... I downloaded it and it works for me or doesn't work for me because of this. If we can get at least that much participation, we can fix it and get it back out, if required.
Thanks, Johnny Hughes
Hi Johnny,
The stuff in the testing repo has already been initially tested (and packaged) by one of the CentOS Developers. It doesn't get in there if it is not at least WorksForMe quality for someone who should understand enterprise ready.
Now I not suggesting that it is perfect, thus the need for testing.
What we need is people to use the products, make sure they work as expected, and tell us it does or does not work for them.
Thanks for the clarification.
We certainly are also open for suggestions and/or packaging comments and better ways to do things.
I would settle for ... I downloaded it and it works for me or doesn't work for me because of this. If we can get at least that much participation, we can fix it and get it back out, if required.
Personally, I'd not object to testing some packages that I use for my workload (like the Sun JDK and PostgreSQL), but I don't actively track packages in testing. It would be much easier if one could 'adopt' a package as a tester, meaning that you will get an e-mail notification when a new package version is released in testing along with the changelog since the last version.
But maybe that's overengineering.
-- Daniel
--- Johnny Hughes mailing-lists@hughesjr.com wrote:
On Fri, 2006-10-13 at 10:01 -0400, danieldk@pobox.com wrote:
Hi Karanbir,
The CentOS Project provides packages in addition
to the packages from
RH's sources. We need to ensure that these
packages are tested in a user
environment for a while, and only released to
the public via the
mirror.centos.org network once there is some
reasonable surety as to the
functional and bugs situation with the package.
I have a question if I may: what should the
packages be tested for?
- Readiness of the packaged software for
enterprise environments.
- Quality of the packaging.
If both, what has most weight? If [1] is
important, I suppose the quality
can partly be tested with regression tests (if the
software includes
regression tests, IMO all enterprise-class
software should). [2]
specifically requires people with enough knowledge
of packaging to go over
it.
Daniel,
The stuff in the testing repo has already been initially tested (and packaged) by one of the CentOS Developers. It doesn't get in there if it is not at least WorksForMe quality for someone who should understand enterprise ready.
Now I not suggesting that it is perfect, thus the need for testing.
What we need is people to use the products, make sure they work as expected, and tell us it does or does not work for them.
point is that peoples usually do not provide feedback, you already know that, they give feedback if:
1- they think the developer think the products are ok and ready (in your case the package is officialy out) 2- are too excited about the product because it solve a great problem 3- they are officialy beta tester (or feel like that) so they feel an obligation to give feedback
I can think another reasons ...
I would settle for ... I downloaded it and it works for me or doesn't work for me because of this. If we can get at least that much participation, we can fix it and get it back out, if required.
I guess a balance approach is to move the packages from testing when: 1- some amount of peoples download the packages from testing _and_ 2- nobody complain about the packages in testing
and then be prepared for more feedback :-) I know this is unwanted but I can't think a way to avoid it because: 1- not all core developers needs or use the packages so not all can fully test the package (unlike what happen when there is only one package or product: linux kernel, apache or CentOS distribution as a whole). 2- users usually do not look into the testing repo, just developers (not only core developers) know where to look, and again not everybody are interested or has the time to test it.
so, finally, my adivice is: release when you, core developer, think is ready and be prepared to re-release after users, and not only developers, give feedback if they do :-)
cu roger
__________________________________________ RedHat Certified Engineer ( RHCE ) Cisco Certified Network Associate ( CCNA )
__________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com
Roger PeXa Escobio wrote:
so, finally, my adivice is: release when you, core developer, think is ready and be prepared to re-release after users, and not only developers, give feedback if they do :-)
we think its ready when there has been sufficient testing been carried out on the packages. Which is what this entire thread was started to be about :)
- KB
Karanbir Singh wrote:
Dear centos-devel'ers, The strategy to release testable rpms to dev.centos.org, then request feedback and move the rpms to stable, when we have enough positive - isnt working. We're not getting that feedback, either positive or negative.
...
So the issue :
The CentOS Project provides packages in addition to the packages from RH's sources. We need to ensure that these packages are tested in a user environment for a while, and only released to the public via the mirror.centos.org network once there is some reasonable surety as to the functional and bugs situation with the package.
Instead of blocking on (lack-of) feedback, I'd suggest considering something like: 1. Put pkgs in "testing" 2. If no bugs reported after X days/weeks, move out of testing
At least this way nothing gets perpetually stalled in testing.
-- Rex
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Rex Dieter schrieb:
Karanbir Singh wrote:
Dear centos-devel'ers, The strategy to release testable rpms to dev.centos.org, then request feedback and move the rpms to stable, when we have enough positive - isnt working. We're not getting that feedback, either positive or negative.
...
So the issue :
Instead of blocking on (lack-of) feedback, I'd suggest considering something like:
- Put pkgs in "testing"
- If no bugs reported after X days/weeks, move out of testing
At least this way nothing gets perpetually stalled in testing.
-- Rex
How about a simple interface to also give positive feedback, perhaps one point why there is not much feedback is that there are not many problems? Maybe a simple tracking system where people can click on "works for me" for each package they use.
Chris
Christoph Maser wrote:
How about a simple interface to also give positive feedback, perhaps one point why there is not much feedback is that there are not many problems? Maybe a simple tracking system where people can click on "works for me" for each package they use.
email works :) just reply to the email posted to this list, announcing availability of the package.
Karanbir Singh wrote as to:
The strategy to release testable rpms to dev.centos.org
On Fri, 13 Oct 2006, Rex Dieter wrote:
Instead of blocking on (lack-of) feedback, I'd suggest considering something like:
- Put pkgs in "testing"
- If no bugs reported after X days/weeks, move out of
testing
At least this way nothing gets perpetually stalled in testing.
Yikes. To torture the truism, 'An absence of evidence is not evidence of an absence' of problems.
Not to put too fine a point on it, but how is automatic promotion out of 'testing' into a chain _desireable_ in an enterprise oriented operating environment?
Clearly some so called 'admin's' will clearly implicitly trust anything (ie., look at the constant traffic into mailing lists for distributions where 'yum' is an available updater with horrific collections of random archives enabled). Why take the reputational risk here?
It may be proper for Red Hat's Fedora, as it has evolved (the firestorms I see regularly erupt on fedora-devel make me doubt this, but ... those participating there without an @redhat.com available to them are all volunteers), but not here. Putting aside stability or security issues, something as simple as added support load makes me want to avoid anything with an 'official' CentOS addon status. The 'Enemies of Carlotta' missed conflict thread I saw today reaffirms my doubt that auto-promotion works based on _assumed_ safety.
My solution, as to my archive of packagings, is simple -- Very general SRPM's exist, and a person who cannot solve a build environment and BuildRequires, (which is documented at my site, along with several other sites which I have contributed to over the years) is probably not going to use my packagings.
When I get a report, I address it. I do not undertake to warrant to any anonymous FTP user, any ongoing (nor even present) security, functionality, or other pedigree to the packagings. Indeed, I have marked certain unsafe ones as I have re-encountered them. This makes the maintenance load manageable.
I have worked on outlines thinking through some of the issues, on building a trustable, and 'vetted' submitted package infrastructure a couple of times. All of the plans fall apart on the relatively low reward for testing compared with the rather high and ongoing load of doing it 'right' and safely.
Before the divergence of cAos and CentOS we were discussing these matters: http://www.herrold.com/caos/QA-requirements.txt and earlier, before Red Hat's takeover of fedora.us, I had posted this into fedora.us's former mailing lists (the former mailing list host: videl.ics.hawaii.edu no longer responds) : http://www.owlriver.com/projects/packaging/fedora-flow.txt [That latter document provoked Warren for what I considered irrational reasons.]
In part CentOS works because it has limited itself to being a relatively strict rebuild effort.
-- Russ Herrold
On Thu, 2006-10-19 at 18:30 -0400, R P Herrold wrote:
Karanbir Singh wrote as to:
The strategy to release testable rpms to dev.centos.org
On Fri, 13 Oct 2006, Rex Dieter wrote:
Instead of blocking on (lack-of) feedback, I'd suggest considering something like:
- Put pkgs in "testing"
- If no bugs reported after X days/weeks, move out of
testing
At least this way nothing gets perpetually stalled in testing.
Yikes. To torture the truism, 'An absence of evidence is not evidence of an absence' of problems.
Not to put too fine a point on it, but how is automatic promotion out of 'testing' into a chain _desireable_ in an enterprise oriented operating environment?
Clearly some so called 'admin's' will clearly implicitly trust anything (ie., look at the constant traffic into mailing lists for distributions where 'yum' is an available updater with horrific collections of random archives enabled). Why take the reputational risk here?
It may be proper for Red Hat's Fedora, as it has evolved (the firestorms I see regularly erupt on fedora-devel make me doubt this, but ... those participating there without an @redhat.com available to them are all volunteers), but not here. Putting aside stability or security issues, something as simple as added support load makes me want to avoid anything with an 'official' CentOS addon status. The 'Enemies of Carlotta' missed conflict thread I saw today reaffirms my doubt that auto-promotion works based on _assumed_ safety.
My solution, as to my archive of packagings, is simple -- Very general SRPM's exist, and a person who cannot solve a build environment and BuildRequires, (which is documented at my site, along with several other sites which I have contributed to over the years) is probably not going to use my packagings.
When I get a report, I address it. I do not undertake to warrant to any anonymous FTP user, any ongoing (nor even present) security, functionality, or other pedigree to the packagings. Indeed, I have marked certain unsafe ones as I have re-encountered them. This makes the maintenance load manageable.
I have worked on outlines thinking through some of the issues, on building a trustable, and 'vetted' submitted package infrastructure a couple of times. All of the plans fall apart on the relatively low reward for testing compared with the rather high and ongoing load of doing it 'right' and safely.
Before the divergence of cAos and CentOS we were discussing these matters: http://www.herrold.com/caos/QA-requirements.txt and earlier, before Red Hat's takeover of fedora.us, I had posted this into fedora.us's former mailing lists (the former mailing list host: videl.ics.hawaii.edu no longer responds) : http://www.owlriver.com/projects/packaging/fedora-flow.txt [That latter document provoked Warren for what I considered irrational reasons.]
In part CentOS works because it has limited itself to being a relatively strict rebuild effort.
Russ,
All that is true ... and for the [base] and [updates] repos I would 100% agree with you.
(As in ... we would not put anything in there that is not 100% vetted, tested, etc.)
However, I disagree concerning the extras and centosplus repos.
These are NOT core packages, there are warnings all over the place stating that, and they are one of the things that distinguishes CentOS from the other rebuild projects. They are "OPT IN" items that have been tested by enterprise users and CentOS Admins and they are important to CentOS users ... at least the ones I know :)
I understand that people can hire consultants or admins who can build these things themselves, however I think that the CentOS project is greatly enhanced as a community project when we provide things like:
Horde, the cenotsplus kernel, maybe the Web Application Stack, etc.
We have had NO ISSUES to date with bad quality items in any of the repos ... and I think everyone understands that the optional CentOS repos are a cut above most others where EL3/4 compatibility and enterprise readiness is concerned.
That is what we are trying to continue to produce.
We certainly do not want to do that to the detriment of the base and updates repo ... nor will anything in the testing repos be put in either base or updates.
Thanks, Johnny Hughes
R P Herrold wrote:
Karanbir Singh wrote as to:
The strategy to release testable rpms to dev.centos.org
On Fri, 13 Oct 2006, Rex Dieter wrote:
Instead of blocking on (lack-of) feedback, I'd suggest considering something like:
- Put pkgs in "testing"
- If no bugs reported after X days/weeks, move out of
testing
At least this way nothing gets perpetually stalled in testing.
Yikes. To torture the truism, 'An absence of evidence is not evidence of an absence' of problems.
I guess I failed to mention that these scheme is what we've used @ kde-redhat repo for ages, and in our experience, it works pretty well and encourages feedback from the userbase. ymmv.
-- Rex
Rex Dieter wrote:
The CentOS Project provides packages in addition to the packages from RH's sources. We need to ensure that these packages are tested in a user environment for a while, and only released to the public via the mirror.centos.org network once there is some reasonable surety as to the functional and bugs situation with the package.
Instead of blocking on (lack-of) feedback, I'd suggest considering something like:
- Put pkgs in "testing"
- If no bugs reported after X days/weeks, move out of testing
At least this way nothing gets perpetually stalled in testing.
we could go one better, and even attach a rule to count number of down loaders. Since dev.centos.org is not mirrored outside that 1 machine we can track the log trivially.
but is that something 'enough' to consider a package releasable ?
Karanbir Singh wrote:
The CentOS Project provides packages in addition to the packages from RH's sources. We need to ensure that these packages are tested in a user environment for a while, and only released to the public via the mirror.centos.org network once there is some reasonable surety as to the functional and bugs situation with the package.
Just a comment on the mysql/php/postgresql rpms in the dev repo. I see Johnny, and I'm sure others, are awaiting the release of the Rehdat Web Application Stack srpms - https://www.redhat.com/archives/nahant-list/2006-October/msg00091.html - Once those are available they apparently will be security patched for 3 years from now - https://www.redhat.com/security/updates/rhappstack/
Does CentOS or someone have a copy of the SRPMS, or know what versions are available? There must be someone on the list here with a subscription...
(My point was maybe to wait on pushing the php, mysql and postgresql rpms from dev into centosplus in case Redhat backports patches to slightly older versions than what are in the dev repo)
Greg
On Fri, 2006-10-13 at 09:15 -0700, Greg Swallow - SkyNet wrote:
Karanbir Singh wrote:
The CentOS Project provides packages in addition to the packages from RH's sources. We need to ensure that these packages are tested in a user environment for a while, and only released to the public via the mirror.centos.org network once there is some reasonable surety as to the functional and bugs situation with the package.
Just a comment on the mysql/php/postgresql rpms in the dev repo. I see Johnny, and I'm sure others, are awaiting the release of the Rehdat Web Application Stack srpms - https://www.redhat.com/archives/nahant-list/2006-October/msg00091.html - Once those are available they apparently will be security patched for 3 years from now - https://www.redhat.com/security/updates/rhappstack/
Does CentOS or someone have a copy of the SRPMS, or know what versions are available? There must be someone on the list here with a subscription...
(My point was maybe to wait on pushing the php, mysql and postgresql rpms from dev into centosplus in case Redhat backports patches to slightly older versions than what are in the dev repo)
Greg,
That is my hope, that we can take all that is in the RHWAS and get that into centosplus ... at least the non jboss app server parts of it that are enterprise maintained SRPMS.
I do believe that these are going to eventually be released on the public RH ftp server ... I am just trying to help the process along a little :-P
Thanks, Johnny Hughes
Johnny Hughes:
I do believe that these are going to eventually be released on the public RH ftp server ... I am just trying to help the process along a little :-P
I am mildly worried. Red Hat Developer Suite has been released on August 30 (according to Red Hat 108), but I haven't seen anything newer than RHDS2.1 on FTP.
-- Daniel
-
Christoph Maser -
danieldk＠pobox.com -
Greg Swallow - SkyNet -
Johnny Hughes -
Karanbir Singh -
R P Herrold -
Rex Dieter -
Roger Pe�a Escobio