As a part of the contrib acceptance process, can we spread the load a bit by saying that anyone who wants to submit a package, must also find someone else ( other than their sponsor ) to write a set of acceptance tests[1] for their package.
I realise that in some cases, its going to be non-trivial writing these sort of tests, eg when looking at a new kernel module that needs specific hardware to be present for it to do something testable. However in many cases its really very trivial ( eg. writing tests for bind are extremely easy, just need to get a set of configs into place and run a dns lookup or a dozen ).
How I imagine this to work would be that a post-build testing harness could do whatever environment/teardown is required, get a clean instance installed, get the fresh packages into there and run the tests. Stage1 could be just a standard test harness that ensures the quality of the rpms being built and how they work as a collection, tools like repomanage etc could be part of Stage1.
Follow that up with the acceptance tests written specifically for the package. Dont care what testing framework is used, even bash will work in most cases ( shunit2 ? ), simpletest, rspec, python-unit. Whatever. As long as it can be called from the command line, it should work.
Just wanted to get the idea out there and see what the general feeling is about this. If people are seriously keen, we cn even rip out unit and integration tests from inside package %build sections and run them post-install :) Although as something to start with, lets not go that far.
[1]: http://en.wikipedia.org/wiki/Acceptance_testing
Dear Karan,
As a part of the contrib acceptance process, can we spread the load a bit by saying that anyone who wants to submit a package, must also find someone else ( other than their sponsor ) to write a set of acceptance tests[1] for their package.
A process like this could easily be handled through package reviews via bucktracker. Maybe there is need of adding a 'review request' category.
A review could then contain a 'quality check' of the spec file (perhaps with Fedora Package Guidelines/rpmlint in mind), followed by a review of provided paches and a rebuild with 'acceptance test'.
Best Regards Marcus
On Thu, Aug 13, 2009 at 4:23 AM, Marcus Moellermail@marcus-moeller.de wrote:
Dear Karan,
As a part of the contrib acceptance process, can we spread the load a bit by saying that anyone who wants to submit a package, must also find someone else ( other than their sponsor ) to write a set of acceptance tests[1] for their package.
A process like this could easily be handled through package reviews via bucktracker. Maybe there is need of adding a 'review request' category.
A review could then contain a 'quality check' of the spec file (perhaps with Fedora Package Guidelines/rpmlint in mind), followed by a review of provided paches and a rebuild with 'acceptance test'.
Of course there will be formal testing done from the testing repo before it makes it into stable.
I think a posting of the spec and it's patches for peer verification and review some place, and have the sponsor verify the source package is legit and that should be enough. Once the spec and patches are verified and final versions agreed upon then they can be pulled right from there to be used in the build process. This should keep everything very transparent, and if someone wants to know what spec and patches were used in building a package they can check out the web-based package review system (PRS).
If bugtracker works, then great, otherwise some other type of web-based system with adequate authentication, version control, approvals, etc.
Maybe a separate wiki type server? Maybe somebody has a better idea?
There's nothing now, so we can think outside the box a little.
Maybe we can have the PRS track packages from submission->testing->stable, so we can see who submitted, who sponsored, who approved, when it went into testing, who signed off on the testing before it reached enough sign-offs for it to make it to stable and so on along with the original source spec, patches and any updates to those.
Does such a beast exist?
-Ross
Hi,
On 08/13/2009 09:23 AM, Marcus Moeller wrote:
As a part of the contrib acceptance process, can we spread the load a bit by saying that anyone who wants to submit a package, must also find someone else ( other than their sponsor ) to write a set of acceptance tests[1] for their package.
A process like this could easily be handled through package reviews via bucktracker. Maybe there is need of adding a 'review request' category.
I am sure we can work out a usable process once we have a target in mind. At the moment, my concern is purely around the idea of writing these tests and what ( if anything ) people had to say about these.
btw, I think a bugtracker is good for tracking bugs. nothing more than that. Tools like mingle, redmine, gitorious etc are much better at handling coding related flow metrics / progress.
Dear Karan,
As a part of the contrib acceptance process, can we spread the load a bit by saying that anyone who wants to submit a package, must also find someone else ( other than their sponsor ) to write a set of acceptance tests[1] for their package.
A process like this could easily be handled through package reviews via bucktracker. Maybe there is need of adding a 'review request' category.
I am sure we can work out a usable process once we have a target in mind. At the moment, my concern is purely around the idea of writing these tests and what ( if anything ) people had to say about these.
btw, I think a bugtracker is good for tracking bugs. nothing more than that. Tools like mingle, redmine, gitorious etc are much better at handling coding related flow metrics / progress.
Fedora is using the bugzilla as bugtracking platform for quite a while without any problems.
Best Regards Marcus
Hi,
On 08/14/2009 07:23 AM, Marcus Moeller wrote:
Fedora is using the bugzilla as bugtracking platform for quite a while without any problems.
As a bug tracking system. bugzilla is quite nice. The only problem I've had ( in the past ) with it was the amount of work required around it and the complexity in setup / admin around it.
The problem I have with the Fedora approach is that once a package is into the system, there is really no audit / tracking or testing of anything binary going through the system.
On 14 Aug 2009, at 12:34 PM, Karanbir Singh wrote:
The problem I have with the Fedora approach is that once a package is into the system, there is really no audit / tracking or testing of anything binary going through the system.
That is not entirely accurate, That is what the karma option is for in the updates system (bondi) while packages are in testing users can critic them even block them from going stable.
On Aug 14, 2009, at 6:34 AM, Karanbir Singh mail-lists@karan.org wrote:
Hi,
On 08/14/2009 07:23 AM, Marcus Moeller wrote:
Fedora is using the bugzilla as bugtracking platform for quite a while without any problems.
As a bug tracking system. bugzilla is quite nice. The only problem I've had ( in the past ) with it was the amount of work required around it and the complexity in setup / admin around it.
The problem I have with the Fedora approach is that once a package is into the system, there is really no audit / tracking or testing of anything binary going through the system.
What we need is a good web-based open-source change management system.
What exists now directly in that category?
If nothing what exists that could be easily made to fit the bill?
I like the openness of wikis, but they lack the audit and approval capabilities, unless someone has developed a mod for a standard one?
-Ross
-
Andrew Colin Kissa -
Karanbir Singh -
Marcus Moeller -
Ross Walker