[CentOS-devel] Scope of testing done by CentOS Development Team

[Somers-Harris, David | David | OPS]

On 01/14/2015 10:58 PM, Karanabir Singh wrote:
> this is something that comes up quite a bit - and it would be good to work on something from 
> the project side so everyone can benefit.
> 
> the common ground is to build up enough of a test scenario that we can ue that to build confidence 
> in external repos, and something that gives the repo admins a target to achieve 'good practice' standards.
> 
> Lets start with the most obvious place : what problem are you looking to solve ?

I want to create a policy for the servers I look after in which we only use the CentOS base and updates repos because we know that the CentOS project has tested that software on CentOS. However, I anticipate that developers will still want to use software which is on other repos like extras and SCL, or even third party repositories like EPEL or Remi, and even have a strong business case for doing so.

For this scenario, I want to prepare a process which if we put the software through, then we can feel confident it will pass muster just as much as the software on base and updates do, if not more. I realize this is a very vague statement since there are various assurances provided by a trustworthy repo.

1. The software is compatible with CentOS
  a. i.e. compatible with all software it touches e.g. if it's a PHP module, it doesn't cause httpd to crash in any expected scenario
2. The software is mature
  a. i.e. stable, secure
3. The software is maintained to ensure the continuity of #2
  a. Either by the repo admin or by the developers themselves.

Since you mentioned this being an opportunity to create a standard for repo admins to aspire after, I guess what I as an operators could ask repo-admins to provide would be

1. Automated tests or test catalogs which I can run on my OS (in this case CentOS).
  a. Standards for what these tests need to include would need to be defined
  b. Obviously the repo-admin would not necessarily be the ones creating this, they can just re-package whatever the developers have already written)
2. A way to distinguish stable, maintained software from untrustworthy software such as bleeding edge or legacy software.
  a. This is an interesting topic actually. Is there a standard method out there for classifying the stability and maintenance lifecycle state of software? If so, that could be used. If not, this could be an opportunity to create one.

Let me know what you think, or if that answers your question at all.