Dne 13.4.2011 8:39, Fabian Arrotin napsal(a):

Hi, Thanks a lot for your submissions, i'll integrate them in the scripts repository.

Fabian, Would you mind opening the process slightly? I mean it'd be helpful having the test list to avoid people doing the same thing. Also it would really help to codify the return values, etc.

One simple remark : we launch the tests on a minimal setup (from a minimal ks) so the first thing to do in the script itself is to install the required package, verify that it installs correctly and then test the feature that you want to test.

This is something that should be written on Wiki page, so that people know: - what are the current scripts - expected return values - testing environment and requirements - etc.

I'm working on redesign quickly something that can be used for the 6.0 tests (now that 5.6 is out of the door)

Fabian

Thanks, DH

On 13/04/11 10:07, David Hrbá?? wrote:

Dne 13.4.2011 8:39, Fabian Arrotin napsal(a):

...

Hi, Thanks a lot for your submissions, i'll integrate them in the scripts repository.

Fabian, Would you mind opening the process slightly? I mean it'd be helpful having the test list to avoid people doing the same thing. Also it would really help to codify the return values, etc.

Hi David,

Sorry for the late answer but due to some illness on my side (and so being very tired when going back home from $work) i wasn't able to answer that question. Yes it's really a need to open that part of the tests so that people can contribute to those tests and also open the QA process (and also the status) to the community

...

One simple remark : we launch the tests on a minimal setup (from a minimal ks) so the first thing to do in the script itself is to install the required package, verify that it installs correctly and then test the feature that you want to test.

This is something that should be written on Wiki page, so that people know:

• what are the current scripts
• expected return values
• testing environment and requirements
• etc.

Yes, and so first a need to design it before documenting it. Let me clarify this a little bit : actually we (Karan and myself) designed quickly something that could be used directly during the 5.6 process (we did that during Fosdem, between discussions with people at the CentOS booth ...) We started 'thin' and so actually it's just a file with different functions declared and some bash scripts (one per VM/type) sourcing that first file and using the functions. Those 'functions' contain CentOS QA hardware informations (like on which machine to deploy a vm, which type of vm (xen, kvm, virtualbox, etc ...) disk-size, memory etc ..

So we'd like first to separate maybe those informations from the tests themselves. The tests scripts would be available in a git repo and we'd use them during QA and post the results somewhere.

I would like to redesign our QA subnet/vlan a little bit and reflect that in our existing QA script. (I don't think that this part interests the Community as it also involves public ip, etc ...)

We'd try to have a small meeting with KB about that part and then a public discussion (an IRC meeting maybe) for people wanting to contribute their tests.

Part of the discussion would be also about what to test. For example, testing if dovecot imap/pop3 is working is good, but my personal approach would be to test if centos packages for example are also 'centosified'. Let me explain that with a real example : if we have to change a logo (and that we use the same logo during the whole distro lifetime) a basic script would be to retrieve that logo, compare it with the known md5sum and so guess if the logo was replaced or not correctly. Stupid example, but believe me it's worth writing such stupid script :-)

Fabian

On Wed, 2011-04-13 at 09:50 +0100, Mister IT Guru wrote:

On Wed, 2011-04-13 at 08:39 +0200, Fabian Arrotin wrote:

...

On 04/13/2011 07:38 AM, Athmane Madjoudj wrote:

...

Hello,

I've attached two test-script for testing IMAP and POP3 services provided by Dovecot (default IMAP/POP3 server in CentOS/RHEL).

Best regards.

Hi,

Thanks a lot for your submissions, i'll integrate them in the scripts repository.

Sorry for jumping into this thread. I'm making it my personal mission to learn as much as I can about CentOS. Please excuse this noob question... What is the purpose of these scripts?

...

One simple remark : we launch the tests on a minimal setup (from a minimal ks) so the first thing to do in the script itself is to install the required package, verify that it installs correctly and then test the feature that you want to test.

The above only further deepens the mystery for me! I want to learn!

...

I'm working on redesign quickly something that can be used for the 6.0 tests (now that 5.6 is out of the door)

Tell me more of these tests! :)

...

Fabian

Hmmm -- I don't mind waiting, I'm trying to hold onto my enthusiasm here. I can guess, or try and piece together an answer for myself, but just some inkling that I was noticed, or something.

On Thu, 2011-04-14 at 10:03 +0100, Michael Simpson wrote:

On 14 April 2011 09:34, Mister IT Guru misteritguru@gmx.com wrote:

...

Hmmm -- I don't mind waiting, I'm trying to hold onto my enthusiasm here. I can guess, or try and piece together an answer for myself, but just some inkling that I was noticed, or something. -- Mister IT Guru

I wondered the same thing so i channeled my inner "misc" and opened up the tests in my favourite text editor.

I understand the phrasing of my question assumed that I hadn't done that - I did the same, and when I noticed the purpose of the scripts, and what they do, I wondered why is this being "repeated"

They test for basic functionality ie create a basic configuration, switch it on and check that it is working and give an exit status of pass or fail.

This I understand - what I would like to know, is this a new thing? Or is this part of the normal procedure? Are these scripts uploaded to a SVN repo where we can get to it, and test against our own builds if we are so inclined? etc etc

The conversation above suggests that it would be gold standard for the test for each service to be able to successfully install the service onto a minimal set up as well rather than having separate install and test scripts.

If this is standard - then I'm surprised that the distribution mechanism is the list. (I'm sure I'm wrong, but understand, this is all I can see as a newbie, or even someone who was just browsing to see how projects are run) - I just assumed that the devs used automated script deployment mechanisms to do this kind of thing.

I'd use nagios as a front end, with NRPE on my target machine with a minimal install, and push the scripts from an SVN repo to the test box, and then look in nagios to see what the results of the script is. That way, I know that when I get green across the board - Everything is all good!

open them up and have a look :)

Way ahead of you! hehe!

I can see what the tests do, I am asking why are we doing these tests?

I'm assuming that this is done as part of the mechanism to test that certain packages are working as expected after install, once they have passed binary compatibility. (I would assume that if they are binary compatible, that they will behave the same way as the RHEL packages) which if this is the case, then why are we testing them again? Which leads me to think my assumption is wrong - SO .... why are we doing these tests? :)

I use nagios to run scripts and get results from environments, along with other deployment methods to create clean environments to do this automagically. Does the centos devs do something like this, (I know I inferred this earlier, bare with me).

Something like this - the writing of the scripts, can that be opened up a bit? Maybe thrown open to the community? Get them cracking on the current release, see what comes back - it's just their time and effort, you just have to ask for it.

Should I be starting a new thread for this by the way, I know how fickle people can be!

regards

mike

Thanks for the response Mike!

On Fri, 2011-04-15 at 06:17 -0500, Johnny Hughes wrote:

The CentOS Project is building a suite of automated tests that we are using to help us with QA after we get our tree ready.

This test suite will run on our tree after we get it moved to our QA servers.

This is being run by people other than Karanbir, Tru, and me ... so it is already "opened up".

As to how it is being run, that is up to the people who we have "opened it up to".

See this here? This should be documented and on the web in a very prominent position. This is awesome to know this.

There are thousands of people who use CentOS who build automated testing suites, using numerous tools and methodologies.

So, so far I've learnt that we use Mock to build the centos packages, and we actually do have a test suite in development.

Great - Now, we just needs those two steps up, online, and for the community to see. Hopefully with hundreds more eyes on it, we can grow even more. Personally, I think I'll start helping you guys with your PR, because getting info feels like getting blood from a stone. Call me impatient of whatever you wish, (not aimed at Johnny Huges or anyone person,) I will continue to pester you guys, and you can hate seeing Mister IT Guru in your inbox!! But I truly believe CentOS needs to come into the light - don't want to CentOS to go the debian root (geddit!)