[CentOS-devel] Scope of testing done by CentOS Development Team

[Johnny Hughes]

On 01/24/2015 08:37 AM, Somers-Harris, David | David | OPS wrote:
> On 01/14/2015 10:58 PM, Karanabir Singh wrote:
>> this is something that comes up quite a bit - and it would be good to work on something from 
>> the project side so everyone can benefit.
>>
>> the common ground is to build up enough of a test scenario that we can ue that to build confidence 
>> in external repos, and something that gives the repo admins a target to achieve 'good practice' standards.
>>
>> Lets start with the most obvious place : what problem are you looking to solve ?
> 
> I want to create a policy for the servers I look after in which we only use the CentOS base and updates repos because we know that the CentOS project has tested that software on CentOS. However, I anticipate that developers will still want to use software which is on other repos like extras and SCL, or even third party repositories like EPEL or Remi, and even have a strong business case for doing so.
> 
> For this scenario, I want to prepare a process which if we put the software through, then we can feel confident it will pass muster just as much as the software on base and updates do, if not more. I realize this is a very vague statement since there are various assurances provided by a trustworthy repo.
> 
> 1. The software is compatible with CentOS
>   a. i.e. compatible with all software it touches e.g. if it's a PHP module, it doesn't cause httpd to crash in any expected scenario
> 2. The software is mature
>   a. i.e. stable, secure
> 3. The software is maintained to ensure the continuity of #2
>   a. Either by the repo admin or by the developers themselves.
> 
> Since you mentioned this being an opportunity to create a standard for repo admins to aspire after, I guess what I as an operators could ask repo-admins to provide would be
> 
> 1. Automated tests or test catalogs which I can run on my OS (in this case CentOS).
>   a. Standards for what these tests need to include would need to be defined
>   b. Obviously the repo-admin would not necessarily be the ones creating this, they can just re-package whatever the developers have already written)
> 2. A way to distinguish stable, maintained software from untrustworthy software such as bleeding edge or legacy software.
>   a. This is an interesting topic actually. Is there a standard method out there for classifying the stability and maintenance lifecycle state of software? If so, that could be used. If not, this could be an opportunity to create one.
> 
> Let me know what you think, or if that answers your question at all.

WRT testing that we do .. basically we do our t_functional tests:

http://wiki.centos.org/QaWiki/AutomatedTests

The purpose of those tests is really to verify that CentOS works like
the upstream packages work.

If we somehow break something in our build that is not broken in the
upstream packages, we will fix it.

However, if the same problem exists in RHEL, we will help find a
solution .. maybe provide a temporary soltuion from a testing location
.. and provide our possible solutions to Red Hat via
bugzilla.redhat.com.  Once Red Hat rolls in the fix, we will then have
it in CentOS.

But back to providing CI .. we (The CentOS Project) are certainly
willing to facilitate the community being able to create tests for all
the CentOS repos via t_functional above.

We are also willing to look at facilitating external software projects
being able to create tests for their software on CentOS and to provide
the ability for those projects to run those tests on hardware we
provide.  We are just in the beginning stages of thinking about this
type of setup, but hopefully we can do something in this area.



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: OpenPGP digital signature
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20150125/f8e1f7bc/attachment.sig>