Hello,
What types of tests does the CentOS Development Team do for software in its repositories?
Where does it get its test catalog from?
I want to create a testing process which is equal to if not more rigorous that the CentOS process for any third party software we use.
For example, if I install the php-mysqlnd package from the Remi repository, I want to test it just as carefully as the php-mysql package has been tested before I get it from the CentOS repository.
If somebody can point me in the right direction I'd appreciate it.
Thanks!
David
P.S. I've moved this question from the centos thread to the centos-devel thread as Sven suggested.
Original post: http://lists.centos.org/pipermail/centos/2015-January/149091.html
On 01/14/2015 01:43 AM, Somers-Harris, David | David | OPS wrote:
Hello,
What types of tests does the CentOS Development Team do for software in its repositories?
Where does it get its test catalog from?
I want to create a testing process which is equal to if not more rigorous that the CentOS process for any third party software we use.
For example, if I install the php-mysqlnd package from the Remi repository, I want to test it just as carefully as the php-mysql package has been tested before I get it from the CentOS repository.
If somebody can point me in the right direction I'd appreciate it.
this is something that comes up quite a bit - and it would be good to work on something from the project side so everyone can benefit.
the common ground is to build up enough of a test scenario that we can ue that to build confidence in external repos, and something that gives the repo admins a target to achieve 'good practice' standards.
Lets start with the most obvious place : what problem are you looking to solve ?
On 01/14/2015 10:58 PM, Karanabir Singh wrote:
this is something that comes up quite a bit - and it would be good to work on something from the project side so everyone can benefit.
the common ground is to build up enough of a test scenario that we can ue that to build confidence in external repos, and something that gives the repo admins a target to achieve 'good practice' standards.
Lets start with the most obvious place : what problem are you looking to solve ?
I want to create a policy for the servers I look after in which we only use the CentOS base and updates repos because we know that the CentOS project has tested that software on CentOS. However, I anticipate that developers will still want to use software which is on other repos like extras and SCL, or even third party repositories like EPEL or Remi, and even have a strong business case for doing so.
For this scenario, I want to prepare a process which if we put the software through, then we can feel confident it will pass muster just as much as the software on base and updates do, if not more. I realize this is a very vague statement since there are various assurances provided by a trustworthy repo.
1. The software is compatible with CentOS a. i.e. compatible with all software it touches e.g. if it's a PHP module, it doesn't cause httpd to crash in any expected scenario 2. The software is mature a. i.e. stable, secure 3. The software is maintained to ensure the continuity of #2 a. Either by the repo admin or by the developers themselves.
Since you mentioned this being an opportunity to create a standard for repo admins to aspire after, I guess what I as an operators could ask repo-admins to provide would be
1. Automated tests or test catalogs which I can run on my OS (in this case CentOS). a. Standards for what these tests need to include would need to be defined b. Obviously the repo-admin would not necessarily be the ones creating this, they can just re-package whatever the developers have already written) 2. A way to distinguish stable, maintained software from untrustworthy software such as bleeding edge or legacy software. a. This is an interesting topic actually. Is there a standard method out there for classifying the stability and maintenance lifecycle state of software? If so, that could be used. If not, this could be an opportunity to create one.
Let me know what you think, or if that answers your question at all.
On 01/24/2015 08:37 AM, Somers-Harris, David | David | OPS wrote:
On 01/14/2015 10:58 PM, Karanabir Singh wrote:
this is something that comes up quite a bit - and it would be good to work on something from the project side so everyone can benefit.
the common ground is to build up enough of a test scenario that we can ue that to build confidence in external repos, and something that gives the repo admins a target to achieve 'good practice' standards.
Lets start with the most obvious place : what problem are you looking to solve ?
I want to create a policy for the servers I look after in which we only use the CentOS base and updates repos because we know that the CentOS project has tested that software on CentOS. However, I anticipate that developers will still want to use software which is on other repos like extras and SCL, or even third party repositories like EPEL or Remi, and even have a strong business case for doing so.
For this scenario, I want to prepare a process which if we put the software through, then we can feel confident it will pass muster just as much as the software on base and updates do, if not more. I realize this is a very vague statement since there are various assurances provided by a trustworthy repo.
- The software is compatible with CentOS
a. i.e. compatible with all software it touches e.g. if it's a PHP module, it doesn't cause httpd to crash in any expected scenario 2. The software is mature a. i.e. stable, secure 3. The software is maintained to ensure the continuity of #2 a. Either by the repo admin or by the developers themselves.
Since you mentioned this being an opportunity to create a standard for repo admins to aspire after, I guess what I as an operators could ask repo-admins to provide would be
- Automated tests or test catalogs which I can run on my OS (in this case CentOS).
a. Standards for what these tests need to include would need to be defined b. Obviously the repo-admin would not necessarily be the ones creating this, they can just re-package whatever the developers have already written) 2. A way to distinguish stable, maintained software from untrustworthy software such as bleeding edge or legacy software. a. This is an interesting topic actually. Is there a standard method out there for classifying the stability and maintenance lifecycle state of software? If so, that could be used. If not, this could be an opportunity to create one.
Let me know what you think, or if that answers your question at all.
WRT testing that we do .. basically we do our t_functional tests:
http://wiki.centos.org/QaWiki/AutomatedTests
The purpose of those tests is really to verify that CentOS works like the upstream packages work.
If we somehow break something in our build that is not broken in the upstream packages, we will fix it.
However, if the same problem exists in RHEL, we will help find a solution .. maybe provide a temporary soltuion from a testing location .. and provide our possible solutions to Red Hat via bugzilla.redhat.com. Once Red Hat rolls in the fix, we will then have it in CentOS.
But back to providing CI .. we (The CentOS Project) are certainly willing to facilitate the community being able to create tests for all the CentOS repos via t_functional above.
We are also willing to look at facilitating external software projects being able to create tests for their software on CentOS and to provide the ability for those projects to run those tests on hardware we provide. We are just in the beginning stages of thinking about this type of setup, but hopefully we can do something in this area.
-
Johnny Hughes -
Karanbir Singh -
Somers-Harris, David | David | OPS