firstly, apologies for the cross post.
I've setup an example to illustrate some very simple things we can do to validate a collection. In this case the vagrant collection.
https://github.com/kbsingh/validate-vagrant-scl has the test script.
It does 2 things: 1) makes sure all the rpms we expect to have in the repo are there. 2) it will install the collection, run a centos/7 vagrant box, and run the centos functional test suite inside the box. if this passes, we can assert that vagrant installed ok, was able to run a libvirt backed box properly, and was able to provide typical expected functionality.
There maybe 100s more features of the collection we are not testing here, but that might not be needed if a simple baseline is established. And this test can be run, from triggers in ci.centos.org, watching the 3 repos we care about for the vagrant collection ( ruby22 / ror41 and vagrant itself ).
thoughts ?