[CentOS-devel] functional testing as a contrib requirement

Thu Aug 13 00:09:28 UTC 2009
Karanbir Singh <mail-lists at karan.org>

As a part of the contrib acceptance process, can we spread the load a 
bit by saying that anyone who wants to submit a package, must also find 
someone else ( other than their sponsor ) to write a set of acceptance 
tests[1] for their package.

I realise that in some cases, its going to be non-trivial writing these 
sort of tests, eg when looking at a new kernel module that needs 
specific hardware to be present for it to do something testable. However 
in many cases its really very trivial ( eg. writing tests for bind are 
extremely easy, just need to get a set of configs into place and run a 
dns lookup or a dozen ).

How I imagine this to work would be that a post-build testing harness 
could do whatever environment/teardown is required, get a clean instance 
installed, get the fresh packages into there and run the tests. Stage1 
could be just a standard test harness that ensures the quality of the 
rpms being built and how they work as a collection, tools like 
repomanage etc could be part of Stage1.

Follow that up with the acceptance tests written specifically for the 
package. Dont care what testing framework is used, even bash will work 
in most cases ( shunit2 ? ), simpletest, rspec, python-unit. Whatever. 
As long as it can be called from the command line, it should work.

Just wanted to get the idea out there and see what the general feeling 
is about this. If people are seriously keen, we cn even rip out unit and 
integration tests from inside package %build sections and run them 
post-install :) Although as something to start with, lets not go that far.

[1]: http://en.wikipedia.org/wiki/Acceptance_testing
Karanbir Singh : http://www.karan.org/  : 2522219 at icq