Hi,
The CentOS project has some reasonable test infra in place that tests for the content inside and the installer / media for CentOS Linux. The big two parts of this currently are :
1) Functional tests for the platform are visible at http://ci.dev.centos.org/ and are always being expanded. More information about the setup this runs on is available at : http://wiki.centos.org/QaWiki/CiDev and details on the t_functonal test suite ( what we consider our acceptance tests ) are at: http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional
2) The platform tests : where we test the distro as a whole, including the installer and delivery media, are run on an old IBM BladeCenter. More info at : http://wiki.centos.org/QaWiki/Hardware
Over the last few weeks, I've been looking into setting up and running a wider community available and driveable test infrastructure that allows projects outside of CentOS Linux to come and test their code via a CI or a CD process against CentOS Linux. The aim being to allow them to constantly integrate within their own environment but also to test against CentOS Linux constantly. Allowing both sides of the equation the ability to deliver confidence and to spot issues arising from changes as early as possible.
As the collection of projects in this mix grows, it would allow them to not only CI against CentOS Linux, but also amongst each other, thereby setting up a level of trust-matrix across evolving code.
Towards this goal, we have some hardware that has come online and we've spent ( its mostly only been Fabian! ) some time playing with the platform and getting used to the setup. The platform is a large SeaMicro integrated chassis that includes 64 physical nodes, that can be controlled via an api to the management unit. Details for the hardware setup are available at : http://wiki.centos.org/QaWiki/PubHardware
Just as a differentiator, we are calling this setup ci.centos.org, where external projects can come test and integrate against CentOS Linux. As opposed to ci.dev.centos.org where we test CentOS Linux itself. The overall aim being that we deliver into ci.centos.org a known good CentOS Linux platform that has already been through a level of trust-buildup. Then being delivered on a platform we trust, we hope that failure reports down to infra or CentOS Linux itself are minimised - creating a higher level of trust in the test-run output for the external projects.
In order to deliver services, the workflow that I've been thinking about is to request folks to ask for access to this infra via bugs.centos.org and provide details on what they want to test and how. Based on that we can allocate either physical or virtual machines in the machine pool, and setup the jenkins jobs + create acls etc allowing the project to get access to manage their jobs. They would then be able to setup the callback triggers from either git.centos.org or from external git repositories and also from CentOS rpms.
The service delivery is run from a couple of servers hosted outside the SeaMicro unit. This is where we would run the jenkins instances, the local mirrors, jump hosts and admin nodes to manage the seamicro instances - allowing for the entire seamicro unit to be available for testing purposes only. diagrams on http://wiki.centos.org/QaWiki/PubHardware should help clear out the exact setup and how this runs.
The layout and process being proposed is based around conversations we've had with a few other projects on what might work best for them in order to come and test with us, however the process and workflow and even components we use in the setup are very much up for conversation and change. One issue recently highlighted was that some projects might not want to use jenkins, and thats ok - we are willing to setup and run something else as needed as well.
From the CentOS Project side of things what we want to focus on is
running the hardware, helping admin the service delivery machines and help projects onramp into this environment - we need the projects to come forward and actually run the tests, consume the results and take actions as needed from fallouts. And this can be either projects associated with CentOS in the SIGs or completely orthogonal third party projects looking to get involved in the larger CentOS ecosystem. We welcome projects that need physical machines to run their tests on, and even those that need clusters of machines.
For the actual test runs, we wont really have persistence in test environments. We will be automating the provisioning and deployment, so that the machines can be reinstalled after the test run is done and reused for other purposes. And we encourage folks to test on baremetal since that means tests should complete faster, and unless it really is a virtualisation technology being tested, you are more likely to be closer in terms of user-facing-test-cases.
I am now soliciting comments, questions and interest from external projects who might be willing to come work with us here. We are in a ready-to-go state here.