[CentOS-devel] [QA] Test-scripts for Dovecot POP3/IMAP

Sat Apr 16 17:07:59 UTC 2011
Fabian Arrotin <fabian.arrotin at arrfab.net>

On 13/04/11 10:07, David Hrbá?? wrote:
> Dne 13.4.2011 8:39, Fabian Arrotin napsal(a):
>> Hi,
>> Thanks a lot for your submissions, i'll integrate them in the scripts
>> repository.
>
> Fabian,
> Would you mind opening the process slightly? I mean it'd be helpful
> having the test list to avoid people doing the same thing. Also it would
> really help to codify the return values, etc.

Hi David,

Sorry for the late answer but due to some illness on my side (and so 
being very tired when going back home from $work) i wasn't able to 
answer that question. Yes it's really a need to open that part of the 
tests so that people can contribute to those tests and also open the QA 
process (and also the status) to the community


>
>> One simple remark : we launch the tests on a minimal setup (from a
>> minimal ks) so the first thing to do in the script itself is to install
>> the required package, verify that it installs correctly and then test
>> the feature that you want to test.
>
> This is something that should be written on Wiki page, so that people know:
> - what are the current scripts
> - expected return values
> - testing environment and requirements
> - etc.

Yes, and so first a need to design it before documenting it. Let me 
clarify this a little bit : actually we (Karan and myself) designed 
quickly something that could be used directly during the 5.6 process (we 
did that during Fosdem, between discussions with people at the CentOS 
booth ...)
We started 'thin' and so actually it's just a file with different 
functions declared and some bash scripts (one per VM/type) sourcing that 
first file and using the functions.
Those 'functions' contain CentOS QA hardware informations (like on which 
machine to deploy a vm, which type of vm (xen, kvm, virtualbox, etc ...) 
disk-size, memory etc ..

So we'd like first to separate maybe those informations from the tests 
themselves. The tests scripts would be available in a git repo and we'd 
use them during QA and post the results somewhere.

I would like to redesign our QA subnet/vlan a little bit and reflect 
that in our existing QA script. (I don't think that this part interests 
the Community as it also involves public ip, etc ...)

We'd try to have a small meeting with KB about that part and then a 
public discussion (an IRC meeting maybe) for people wanting to 
contribute their tests.

Part of the discussion would be also about what to test. For example, 
testing if dovecot imap/pop3 is working is good, but my personal 
approach would be to test if centos packages for example are also 
'centosified'.
Let me explain that with a real example : if we have to change a logo 
(and that we use the same logo during the whole distro lifetime) a basic 
script would be to retrieve that logo, compare it with the known md5sum 
and so guess if the logo was replaced or not correctly. Stupid example, 
but believe me it's worth writing such stupid script :-)

Fabian