[CentOS-devel] a post-build test harness

[Karanbir Singh]

Jeff Johnson wrote:
> What is "Rspec"? I'm unfamiliar with the reference.

Essentially a unit testing framework that works on specifications rather 
than tests. Let me see if I can come up with a semi functional example 
to demonstrate what I have in mind.

> Per-pkg unit tests are likely best done by running rpmlint with a custom 
> configuration.

I've looked at rpmlint a few times, let me look again. Perhaps most of 
this might be plumbable via there.

> That introduces a 3rd layer, in addition to unit & integration testing, 
> checking
> that indeed, a package appears in distribution URI's as intended.

I think some / most of this info might be extractable from createrep'd 
metadata. Will need to dig more into that, at the moment just looking at 
individual package testing, and some collection stuff. Eg. making sure 
we get the right multilib stuff into the right place.

> One way to avoid the test for higher is to automate R++ when building.
> The problem these days is how to recognize templates for Release: so
> that an autoincrement might be performed reliably. I suspect that
> EVR autoincrement is an easier problem to solve than detecting
> failure after the build.

I agree with the auto bump, or check at the buildsystem level to make 
sure that package-to-build is indeed higher. Its whats in place at the 
moment here. However, its not easy to enforce on the main distro level 
stuff, since we need to 'inhert' Rel: Ver: from upstream and match it 
exactly1[]

>> And having this harness pass or fail depending on the test outcome.
>>
> 
> The "fail" case is often painful, requiring human intervention.
> A more automated "fail" proof process flow is perhaps a better
> approach than a "fail" test.
> 
> OTOH, detection is better than not testing at all, even if automated 
> correction
> or prevention (through processes) is better than "fail" detection.

I think you put it better than I was able to, the whole aim is to catch 
these things before it breaks or causes issues on production machines 
out there. We dont really churn out that many packages per day that 
would make such reporting get noisy and irritating, as yet.


- KB

[1]: except packages we change, those get a .centos in the Release tag 
and we can control it using homebrew scripting.