Jeff Johnson wrote:
What is "Rspec"? I'm unfamiliar with the reference.
Essentially a unit testing framework that works on specifications rather than tests. Let me see if I can come up with a semi functional example to demonstrate what I have in mind.
Per-pkg unit tests are likely best done by running rpmlint with a custom configuration.
I've looked at rpmlint a few times, let me look again. Perhaps most of this might be plumbable via there.
That introduces a 3rd layer, in addition to unit & integration testing, checking that indeed, a package appears in distribution URI's as intended.
I think some / most of this info might be extractable from createrep'd metadata. Will need to dig more into that, at the moment just looking at individual package testing, and some collection stuff. Eg. making sure we get the right multilib stuff into the right place.
One way to avoid the test for higher is to automate R++ when building. The problem these days is how to recognize templates for Release: so that an autoincrement might be performed reliably. I suspect that EVR autoincrement is an easier problem to solve than detecting failure after the build.
I agree with the auto bump, or check at the buildsystem level to make sure that package-to-build is indeed higher. Its whats in place at the moment here. However, its not easy to enforce on the main distro level stuff, since we need to 'inhert' Rel: Ver: from upstream and match it exactly1[]
And having this harness pass or fail depending on the test outcome.
The "fail" case is often painful, requiring human intervention. A more automated "fail" proof process flow is perhaps a better approach than a "fail" test.
OTOH, detection is better than not testing at all, even if automated correction or prevention (through processes) is better than "fail" detection.
I think you put it better than I was able to, the whole aim is to catch these things before it breaks or causes issues on production machines out there. We dont really churn out that many packages per day that would make such reporting get noisy and irritating, as yet.
- KB
[1]: except packages we change, those get a .centos in the Release tag and we can control it using homebrew scripting.