[CentOS-devel] what / how could test reports be published

[Karanbir Singh]

Hi,

On 04/06/2011 01:01 PM, Thomas Bendler wrote:
>     [... automated VM install ...]
> in principle a good idea, should be (from my point of view) extended to

It works in practise as well :) ( over 700 automated installs during the 
centos-5.6 test phase would say so .. )

> something like BFO (http://boot.fedoraproject.org/) to make it possible
> to install on physical machines as well. So something like USB boot
> stick which get standard configurations based on kickstart files, do the
> installation, report back to console and central QA server and do the
> next install on the list (the only question is how to make previous runs
> persistent so installations won't be done twice until QA system request
> a second install because of a failure).

It would be good to get boot.k.o functionality included in, but afaik 
that needs httpfs etc into the kernel; so we might only be able to 
achieve that for the centosplus kernel. Would you be happy to take on 
the task of looking into whats involved and how feasible that might be ?

The idea of getting real-iron into the QA testing loop is very 
interesting and perhaps even essential. Could a cobbler type setup, 
being able to do ipmi calls to real iron be a good substitute ?

> - lshw

there is no such thing on CentOS

> - dmesg

added.

> - maybe bootchart

not sure about this. Maybe bootchart makes for a good test case in itself ?

> GIT is a good idea but you need a wrapper around that check successful
> runs and shorten this one to profile xyz on hardware abc has no problem.

the test run script does that already, so it will output something like 
this : (taken from a real test ):

78 test cases complete.  73 passes, 5 fails and 0 exceptions.

> See above, an USB stick for real hardware and an ISO for virtual
> hardware will probably work best.

Ideally, we should be testing every install type ( so from boot.iso, 
over pxe + ftp, pxe + http, pxe + images, cd1, dvd, diskimage etc ). At 
the moment though its only usrin virt-install ( so pxe images ) into Xen 
hosts using installmethod = http.

I guess if someone wants to propose automation for the various things it 
would be good. Is it even possible to emulate a usb disk somewhere ( 
kind of like being able to use an iso image as a cd-rom in xen/kvm etc )

- KB