[CentOS-devel] a post-build test harness

[Jeff Johnson]

On Nov 28, 2008, at 10:56 AM, Karanbir Singh wrote:

> Jeff Johnson wrote:
>> What is "Rspec"? I'm unfamiliar with the reference.
>
> Essentially a unit testing framework that works on specifications  
> rather than tests. Let me see if I can come up with a semi  
> functional example to demonstrate what I have in mind.
>

Yes, I found Rspec soon after I asked.

The Rspec agile approach, populating the desired test framework at  
high level *even if the tests do nothing*
is dead-on imho. What tends to happen with testing is that interest  
wanes, and round 'tuits go missing, so
tests that are incrementally added to a framework often don't  
accomplish what was originally intended (in my experience).

>> Per-pkg unit tests are likely best done by running rpmlint with a  
>> custom configuration.
>
> I've looked at rpmlint a few times, let me look again. Perhaps most  
> of this might be plumbable via there.
>

The trick with rpmlint, like all lints, is to actively disable the  
spewage until the fire hose
is merely a trickle. Then proceed by turning on whatever looks useful.  
If you expect rpmlint
*by default* to find all the errors, well, your definition of "error"  
is likely going to
be different thatn rpmlint's.

I watched RedHat QA struggle with Mandrake packaging policies applied  
to a RedHat distro
for a year when rpmlint was introduced. Somehow the QA trolls never  
could grasp the insanity
of Mandrake != RedHat packaging policies, and so using rpmlint was  
eventually abandoned.

An agile Rspec approach enables the QA trolls to actively control the  
testing goal rather
than reactively trying to assign a priority to various flaws.

>> That introduces a 3rd layer, in addition to unit & integration  
>> testing, checking
>> that indeed, a package appears in distribution URI's as intended.
>
> I think some / most of this info might be extractable from  
> createrep'd metadata. Will need to dig more into that, at the moment  
> just looking at individual package testing, and some collection  
> stuff. Eg. making sure we get the right multilib stuff into the  
> right place.
>

Some of the tests you wish cannot be performed by looking at single
packages, only by looking at the collection of package metadata.

The metadata aggregation can be done by createrepo, but the XML  
generated by createrepo discards
information from *.rpm packages, and adds additional information from  
comps.

Whether that is a good thing (or not) depends on one's testing goals.  
E.g. I'm
not all sure why half of the items on the proposed verifytree feature  
bullet list are
there. Obsoletes: loops? Self-referential dependencies? Whatever ...

An alternative to repo metadata might be just to import *complete*  
metadata
into your own database. A database, not XML, is more useful for QA  
imho. See
sophie.zarb.org for a very intelligent postgres database that has  
*complete*
*.rpm metadata, where queries to index additional tag data can be  
retrofitted, and
even remote queries or web display, can all be rather easily done.

All you can do with XML is download the next generation and scratch your
cornea all over again with the pointy angle brackets. ;-)

>> One way to avoid the test for higher is to automate R++ when  
>> building.
>> The problem these days is how to recognize templates for Release: so
>> that an autoincrement might be performed reliably. I suspect that
>> EVR autoincrement is an easier problem to solve than detecting
>> failure after the build.
>
> I agree with the auto bump, or check at the buildsystem level to  
> make sure that package-to-build is indeed higher. Its whats in place  
> at the moment here. However, its not easy to enforce on the main  
> distro level stuff, since we need to 'inhert' Rel: Ver: from  
> upstream and match it exactly1[]
>

Sure its easy to enforce, what sort of clue bat do you want added to  
rpmlib? ;-)

Why do you want *exactly* the same Release:? That's much like RedHat  
QA trolls
being cornfused by Mandrake packaging policies imho. If all the patches
are included, and the buildsys is configured identically for a CentOS  
rebuild,
does it really matter what value is in the Release: tag? And if it  
*does* matter
to CentOS, then adding a substitute for buildrelease tracking is  
likely necessary.

I f the package is rebuilt, then the Release: should be bumped alwaysl  
even if
nothing in the package changed, the build system and build tools are  
different,
and its really hard to find issues with two apparently identical  
packages that
behave differently.

>>> And having this harness pass or fail depending on the test outcome.
>>>
>> The "fail" case is often painful, requiring human intervention.
>> A more automated "fail" proof process flow is perhaps a better
>> approach than a "fail" test.
>> OTOH, detection is better than not testing at all, even if  
>> automated correction
>> or prevention (through processes) is better than "fail" detection.
>
> I think you put it better than I was able to, the whole aim is to  
> catch these things before it breaks or causes issues on production  
> machines out there. We dont really churn out that many packages per  
> day that would make such reporting get noisy and irritating, as yet.
>

Could you post a pointer to your current CentOS QA scripts please? I've
likely asked before, but have lost the memories.

73 de Jeff