[CentOS-devel] Testing Nightly and Status Board

[Madhurranjan Mohaan]

Hi Karanbir,

Thanks for the email . Some initial thoughts.

*Tests*:
1. No. of commits across the test suite. Simple things like git log --since
< > can be pumped into graphite . You'll get an understanding of how many
people are committing and how many requests are pending .
2. test suite runs - No of tests that fail in versions 5 and 6 and across
several of the setups you have. This can be just the number .
3. We could one per package and details of that
Eg, package.httpd.tests.success - 4
      package.httpd.tests.failure - 1
4. Other metrics that you need to identify stability of test suites .

I have a few things around querying below ( either csv or json )

*Deploys*:
1. Success/failures based on platforms and mode of deploy.
Example: You can dump simple metrics back into graphite with 1 indicating
success and 0 failure and create a hierarchy as follows: ( This is just an
example)

deploy.kvm.64bit.http.success
deploy.kvm.64bit.nfs.success
deploy.xen.32bit.http.success
etc

and we could write queries to get the Json api to see how deploys across
kvm(Eg. deploy.kvm.*.*.success)  machines have happened over X days and
maybe alert via Zabbix based on what we think it suitable. Graphite would
just chart it and we'll have to have something sitting on top of it but
having said that sending data to graphite( with statsd on top) is simple
which is a point to consider.

We've used this behavior for chef client runs across 200 odd machines at
work and its worked really well for us.

We can have slightly more fancy UI with graphene showing metrics of your
choice:
http://jondot.github.com/graphene/

Mamu

ps: I briefly read about stash board after you wrote about it but have not
tried it .


On Fri, Mar 15, 2013 at 12:05 AM, Karanbir Singh <mail-lists at karan.org>wrote:

> hi mamu,
>
> On 03/13/2013 06:05 PM, Madhurranjan Mohaan wrote:
> > Stashboard sounds good . We could also post messages to a graphite
> instance
> > based on what we metrics we need and it'll give you nice graphs
> ,especially
> > when you want to compare it with other metrics (in case they are
> > comparable). +1 for the idea.
>
> I had a poke around stashboard, and its pretty complicated and
> cumbersome if we want to run it offline (ie. not on appengine ); there
> are some other options as well.
>
> Having seen the deploy + test scenarious we run, what sort of metrics do
> you think we should shovel graphite way ?
>
> --
> Karanbir Singh
> +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
> GnuPG Key : http://www.karan.org/publickey.asc
> _______________________________________________
> CentOS-devel mailing list
> CentOS-devel at centos.org
> http://lists.centos.org/mailman/listinfo/centos-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.centos.org/pipermail/centos-devel/attachments/20130315/fa15c976/attachment.html>