[CentOS-devel] Community test infrastructure

[Karanbir Singh]

Hi,

The CentOS project has some reasonable test infra in place that tests
for the content inside and the installer / media for CentOS Linux. The
big two parts of this currently are :

1) Functional tests for the platform are visible at
http://ci.dev.centos.org/ and are always being expanded. More
information about the setup this runs on is available at :
http://wiki.centos.org/QaWiki/CiDev and details on the t_functonal test
suite ( what we consider our acceptance tests ) are at:
http://wiki.centos.org/QaWiki/AutomatedTests/WritingTests/t_functional

2) The platform tests : where we test the distro as a whole, including
the installer and delivery media, are run on an old IBM BladeCenter.
More info at : http://wiki.centos.org/QaWiki/Hardware

Over the last few weeks, I've been looking into setting up and running a
wider community available and driveable test infrastructure that allows
projects outside of CentOS Linux to come and test their code via a CI or
a CD process against CentOS Linux. The aim being to allow them to
constantly integrate within their own environment but also to test
against CentOS Linux constantly. Allowing both sides of the equation the
ability to deliver confidence and to spot issues arising from changes as
early as possible.

As the collection of projects in this mix grows, it would allow them to
not only CI against CentOS Linux, but also amongst each other, thereby
setting up a level of trust-matrix across evolving code.

Towards this goal, we have some hardware that has come online and we've
spent ( its mostly only been Fabian! ) some time playing with the
platform and getting used to the setup. The platform is a large SeaMicro
integrated chassis that includes 64 physical nodes, that can be
controlled via an api to the management unit. Details for the hardware
setup are available at : http://wiki.centos.org/QaWiki/PubHardware

Just as a differentiator, we are calling this setup ci.centos.org, where
external projects can come test and integrate against CentOS Linux. As
opposed to ci.dev.centos.org where we test CentOS Linux itself. The
overall aim being that we deliver into ci.centos.org a known good CentOS
Linux platform that has already been through a level of trust-buildup.
Then being delivered on a platform we trust, we hope that failure
reports down to infra or CentOS Linux itself are minimised - creating a
higher level of trust in the test-run output for the external projects.

In order to deliver services, the workflow that I've been thinking about
is to request folks to ask for access to this infra via bugs.centos.org
and provide details on what they want to test and how. Based on that we
can allocate either physical or virtual machines in the machine pool,
and setup the jenkins jobs + create acls etc allowing the project to get
access to manage their jobs. They would then be able to setup the
callback triggers from either git.centos.org or from external git
repositories and also from CentOS rpms.

The service delivery is run from a couple of servers hosted outside the
SeaMicro unit. This is where we would run the jenkins instances, the
local mirrors, jump hosts and admin nodes to manage the seamicro
instances - allowing for the entire seamicro unit to be available for
testing purposes only. diagrams on
http://wiki.centos.org/QaWiki/PubHardware should help clear out the
exact setup and how this runs.

The layout and process being proposed is based around conversations
we've had with a few other projects on what might work best for them in
order to come and test with us, however the process and workflow and
even components we use in the setup are very much up for conversation
and change. One issue recently highlighted was that some projects might
not want to use jenkins, and thats ok - we are willing to setup and run
something else as needed as well.

>From the CentOS Project side of things what we want to focus on is
running the hardware, helping admin the service delivery machines and
help projects onramp into this environment - we need the projects to
come forward and actually run the tests, consume the results and take
actions as needed from fallouts. And this can be either projects
associated with CentOS in the SIGs or completely orthogonal third party
projects looking to get involved in the larger CentOS ecosystem. We
welcome projects that need physical machines to run their tests on, and
even those that need clusters of machines.

For the actual test runs, we wont really have persistence in test
environments. We will be automating the provisioning and deployment, so
that the machines can be reinstalled after the test run is done and
reused for other purposes. And we encourage folks to test on baremetal
since that means tests should complete faster, and unless it really is a
virtualisation technology being tested, you are more likely to be closer
in terms of user-facing-test-cases.

I am now soliciting comments, questions and interest from external
projects who might be willing to come work with us here. We are in a
ready-to-go state here.

-- 
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc