Hi,
One of the things that I'd like to get working sometime is a proper post-build test harness for rpm's. At the moment, what we do is quite basic, along with rpmdiff and some comparison tools we do a string of bash-written script tests. There must be another way of doing this, a sort of rpm::unit, for the lack of a better name, but modelled on Rspec. To illustrate :
For Pkg in PackageList: { ensure Pkg.SIGGPG => present; };
Makes sure that Packages we are testing are signed, I guess it could also be ensure=> 'blahblah'; to make sure its signed with a specific key.
A more complex test might be to make sure that content in multilib pkgs match for overlapping content + multilib acceptable content.
An even more complex test might be to make sure any rpm that drops an init script also has a chkconfig stanza in the %post / %pre ( as an example ).
The first step would, of course, be to look at doing this per package, and then growing it to work with multiple packages ( eg. doing things like :
TestPackage: { ensure latest in -> http://mirror.centos.org/centos/updates/i386/ } ; to make sure that this new package does indeed provide an higher EVR than whatever is in the repo at the end of that url.
And having this harness pass or fail depending on the test outcome.
Perhaps this is not the right list, but most of the people who would end up using this or having to live with me forcing them to use it are on this list... so I thought this would be a good place to start.
Comments ? Am I wasting my time ?
- KB
On Wed, 26 Nov 2008, Karanbir Singh wrote:
Hi,
One of the things that I'd like to get working sometime is a proper post-build test harness for rpm's. At the moment, what we do is quite basic, along with rpmdiff and some comparison tools we do a string of bash-written script tests. There must be another way of doing this, a sort of rpm::unit, for the lack of a better name, but modelled on Rspec. To illustrate :
For Pkg in PackageList: { ensure Pkg.SIGGPG => present; };
Makes sure that Packages we are testing are signed, I guess it could also be ensure=> 'blahblah'; to make sure its signed with a specific key.
A more complex test might be to make sure that content in multilib pkgs match for overlapping content + multilib acceptable content.
An even more complex test might be to make sure any rpm that drops an init script also has a chkconfig stanza in the %post / %pre ( as an example ).
The first step would, of course, be to look at doing this per package, and then growing it to work with multiple packages ( eg. doing things like :
TestPackage: { ensure latest in -> http://mirror.centos.org/centos/updates/i386/ } ; to make sure that this new package does indeed provide an higher EVR than whatever is in the repo at the end of that url.
And having this harness pass or fail depending on the test outcome.
Perhaps this is not the right list, but most of the people who would end up using this or having to live with me forcing them to use it are on this list... so I thought this would be a good place to start.
Comments ? Am I wasting my time ?
I'm working on a tool for doing some of these same things for fedora right now.
Here's the list of things we wanted to check in fedora for a tree of pkgs:
#~ Checks: #~ Package/repo checks: #~ metadata matches #~ comps correct #~ file conflicts #~ normal conflicts #~ obsoleted pkgs in tree #~ circular obsoletes #~ self obsoleting pkgs #~ unresolveable provides #~ Tree/Distro #~ verify all files in .treeinfo exist/match #~ check for sha1sums if it finds .iso files
#~ Notifications: #~ Package: #~ report all triggers #~ look for all postrans/pretrans things #~ self-provided filedeps #~ self-provided normal deps #~ soname provides conflicts #~ in general list out all multiple providers
I'm doing the work in the 'verifytree' tool in yum-utils.
will definitely work on centos 5.3 :)
-sv
On Nov 26, 2008, at 12:45 PM, Karanbir Singh wrote:
Hi,
One of the things that I'd like to get working sometime is a proper post-build test harness for rpm's. At the moment, what we do is quite basic, along with rpmdiff and some comparison tools we do a string of bash-written script tests. There must be another way of doing this, a sort of rpm::unit, for the lack of a better name, but modelled on Rspec. To illustrate :
What is "Rspec"? I'm unfamiliar with the reference.
For Pkg in PackageList: { ensure Pkg.SIGGPG => present; };
Makes sure that Packages we are testing are signed, I guess it could also be ensure=> 'blahblah'; to make sure its signed with a specific key.
A more complex test might be to make sure that content in multilib pkgs match for overlapping content + multilib acceptable content.
An even more complex test might be to make sure any rpm that drops an init script also has a chkconfig stanza in the %post / %pre ( as an example ).
The first step would, of course, be to look at doing this per package, and then growing it to work with multiple packages ( eg. doing things like :
Per-pkg unit tests are likely best done by running rpmlint with a custom configuration.
Multi-pkg tests require collecting an approximation to "everything" that is currently in the distro. How the collection is done, and whether the picture of "everything" is accurate, is (or has been) what has prevented better distro QA imho.
TestPackage: { ensure latest in -> http://mirror.centos.org/centos/updates/i386/ } ; to make sure that this new package does indeed provide an higher EVR than whatever is in the repo at the end of that url.
That introduces a 3rd layer, in addition to unit & integration testing, checking that indeed, a package appears in distribution URI's as intended.
One way to avoid the test for higher is to automate R++ when building. The problem these days is how to recognize templates for Release: so that an autoincrement might be performed reliably. I suspect that EVR autoincrement is an easier problem to solve than detecting failure after the build.
And having this harness pass or fail depending on the test outcome.
The "fail" case is often painful, requiring human intervention. A more automated "fail" proof process flow is perhaps a better approach than a "fail" test.
OTOH, detection is better than not testing at all, even if automated correction or prevention (through processes) is better than "fail" detection.
Perhaps this is not the right list, but most of the people who would end up using this or having to live with me forcing them to use it are on this list... so I thought this would be a good place to start.
Comments ? Am I wasting my time ?
Doesn't sound like a waste of time to me.
73 de Jeff
- KB
CentOS-devel mailing list CentOS-devel@centos.org http://lists.centos.org/mailman/listinfo/centos-devel
Jeff Johnson wrote:
What is "Rspec"? I'm unfamiliar with the reference.
Essentially a unit testing framework that works on specifications rather than tests. Let me see if I can come up with a semi functional example to demonstrate what I have in mind.
Per-pkg unit tests are likely best done by running rpmlint with a custom configuration.
I've looked at rpmlint a few times, let me look again. Perhaps most of this might be plumbable via there.
That introduces a 3rd layer, in addition to unit & integration testing, checking that indeed, a package appears in distribution URI's as intended.
I think some / most of this info might be extractable from createrep'd metadata. Will need to dig more into that, at the moment just looking at individual package testing, and some collection stuff. Eg. making sure we get the right multilib stuff into the right place.
One way to avoid the test for higher is to automate R++ when building. The problem these days is how to recognize templates for Release: so that an autoincrement might be performed reliably. I suspect that EVR autoincrement is an easier problem to solve than detecting failure after the build.
I agree with the auto bump, or check at the buildsystem level to make sure that package-to-build is indeed higher. Its whats in place at the moment here. However, its not easy to enforce on the main distro level stuff, since we need to 'inhert' Rel: Ver: from upstream and match it exactly1[]
And having this harness pass or fail depending on the test outcome.
The "fail" case is often painful, requiring human intervention. A more automated "fail" proof process flow is perhaps a better approach than a "fail" test.
OTOH, detection is better than not testing at all, even if automated correction or prevention (through processes) is better than "fail" detection.
I think you put it better than I was able to, the whole aim is to catch these things before it breaks or causes issues on production machines out there. We dont really churn out that many packages per day that would make such reporting get noisy and irritating, as yet.
- KB
[1]: except packages we change, those get a .centos in the Release tag and we can control it using homebrew scripting.
On Nov 28, 2008, at 10:56 AM, Karanbir Singh wrote:
Jeff Johnson wrote:
What is "Rspec"? I'm unfamiliar with the reference.
Essentially a unit testing framework that works on specifications rather than tests. Let me see if I can come up with a semi functional example to demonstrate what I have in mind.
Yes, I found Rspec soon after I asked.
The Rspec agile approach, populating the desired test framework at high level *even if the tests do nothing* is dead-on imho. What tends to happen with testing is that interest wanes, and round 'tuits go missing, so tests that are incrementally added to a framework often don't accomplish what was originally intended (in my experience).
Per-pkg unit tests are likely best done by running rpmlint with a custom configuration.
I've looked at rpmlint a few times, let me look again. Perhaps most of this might be plumbable via there.
The trick with rpmlint, like all lints, is to actively disable the spewage until the fire hose is merely a trickle. Then proceed by turning on whatever looks useful. If you expect rpmlint *by default* to find all the errors, well, your definition of "error" is likely going to be different thatn rpmlint's.
I watched RedHat QA struggle with Mandrake packaging policies applied to a RedHat distro for a year when rpmlint was introduced. Somehow the QA trolls never could grasp the insanity of Mandrake != RedHat packaging policies, and so using rpmlint was eventually abandoned.
An agile Rspec approach enables the QA trolls to actively control the testing goal rather than reactively trying to assign a priority to various flaws.
That introduces a 3rd layer, in addition to unit & integration testing, checking that indeed, a package appears in distribution URI's as intended.
I think some / most of this info might be extractable from createrep'd metadata. Will need to dig more into that, at the moment just looking at individual package testing, and some collection stuff. Eg. making sure we get the right multilib stuff into the right place.
Some of the tests you wish cannot be performed by looking at single packages, only by looking at the collection of package metadata.
The metadata aggregation can be done by createrepo, but the XML generated by createrepo discards information from *.rpm packages, and adds additional information from comps.
Whether that is a good thing (or not) depends on one's testing goals. E.g. I'm not all sure why half of the items on the proposed verifytree feature bullet list are there. Obsoletes: loops? Self-referential dependencies? Whatever ...
An alternative to repo metadata might be just to import *complete* metadata into your own database. A database, not XML, is more useful for QA imho. See sophie.zarb.org for a very intelligent postgres database that has *complete* *.rpm metadata, where queries to index additional tag data can be retrofitted, and even remote queries or web display, can all be rather easily done.
All you can do with XML is download the next generation and scratch your cornea all over again with the pointy angle brackets. ;-)
One way to avoid the test for higher is to automate R++ when building. The problem these days is how to recognize templates for Release: so that an autoincrement might be performed reliably. I suspect that EVR autoincrement is an easier problem to solve than detecting failure after the build.
I agree with the auto bump, or check at the buildsystem level to make sure that package-to-build is indeed higher. Its whats in place at the moment here. However, its not easy to enforce on the main distro level stuff, since we need to 'inhert' Rel: Ver: from upstream and match it exactly1[]
Sure its easy to enforce, what sort of clue bat do you want added to rpmlib? ;-)
Why do you want *exactly* the same Release:? That's much like RedHat QA trolls being cornfused by Mandrake packaging policies imho. If all the patches are included, and the buildsys is configured identically for a CentOS rebuild, does it really matter what value is in the Release: tag? And if it *does* matter to CentOS, then adding a substitute for buildrelease tracking is likely necessary.
I f the package is rebuilt, then the Release: should be bumped alwaysl even if nothing in the package changed, the build system and build tools are different, and its really hard to find issues with two apparently identical packages that behave differently.
And having this harness pass or fail depending on the test outcome.
The "fail" case is often painful, requiring human intervention. A more automated "fail" proof process flow is perhaps a better approach than a "fail" test. OTOH, detection is better than not testing at all, even if automated correction or prevention (through processes) is better than "fail" detection.
I think you put it better than I was able to, the whole aim is to catch these things before it breaks or causes issues on production machines out there. We dont really churn out that many packages per day that would make such reporting get noisy and irritating, as yet.
Could you post a pointer to your current CentOS QA scripts please? I've likely asked before, but have lost the memories.
73 de Jeff
-
Jeff Johnson -
Karanbir Singh -
Seth Vidal