[Cdash] Newbie questions

[Julien Jomier]

Hi Jeff,

Welcome to CDash! (and these are not bad questions :)).

CDash is really meant to be generic and we know some users are not using 
CMake/CTest to do the submission of the project. We actually have used 
CDash to report testing from PHP applications and we use it on a regular 
basis to test our digital archiving system (MIDAS). So I think it's 
definitely doable, as long as we can translate the report to an XML 
format that CDash would understand. We are also open to add new features 
that would be missing from CDash.

> Is the submission XML schema documented anywhere?  Are there any guarantees about the stability of this XML scheme between versions of CDash?

The documentation regarding the schema is located here:

   http://public.kitware.com/Wiki/CDash:XML

It is more a validation schema than a full documentation but it should 
get you started. Don't hesitate if you have any questions. We cannot 
guarantee that the XML schema won't change, because we add new tags when 
we need it in order to improve the system. That said, we rarely remove 
tags (we actually never did). The base XML schema hasn't change in the 
last seven years or so (taken from Dart, the predecessor of CDash).

> 1. We test distribution tarballs, not SVN checkouts.  Is that against the philosophy of CDash?

Definitely not, any kind of testing is welcome!

> 2. A major theme of our web reporter is aggregation of data, and a drill-down philosophy to find the precise data that you're looking for.  For example, on a good night, we might have *many* of what CDash calls "Configure" executions.  We like to group these together by various attributes (e.g., tarball version, organization, architecture, compiler suite, configure option set, ...etc.) and then allow a web user to drill down to the "Configure" executions that they want to see/examine.
> You can see this in the URL that I cited above; every item in the results table is clickable.  Clicking on a result table narrows the query to similar results.  You keep drilling down until you find just the result set that you want, and then you click on the "Details" button to see the gory details.
> Is aggregation like this possible in CDash?

This is interesting. CDash has this notion of "tracks/groups" which 
allows to group "builds" together so that would be an option (maybe not 
ideal though). The other option would be to come up with a new view of 
the data with a grouping concept. This is something we would have to 
investigate.

> 3. What is the philosophy of contribution to CDash?  I see that it's open source and the SVN is available, but all of the contributors are listed as being at Kitware (http://www.cdash.org/cdash/project/parti.html).  Are you open to code contributions from elsewhere?  (i.e., if we buy into CDash, we may want to submit stuff upstream)

We are always welcoming contributions from other people. We have 
actually integrated many feature requests submitted by developers via 
the bug tracker. We have to be careful with the contributions to CDash 
(as well as CMake/CTest) because it's affecting a lot of projects. We 
are usually quick at integrating patches and bug fixes submitted to us 
after complete review. We should probably upgrade this page with the 
authors of the patches (thanks for pointing this out).

> 4. CDash seems to define 3 test outcoming: Pass, Fail, NotRun (MTT calls this "Skip").  MTT also has a "Timeout" result -- meaning that the test didn't meet the criteria for passing or failing; it was still running when a timeout expired (so the MTT client killed it and moved on to the next test).  Is it possible to extend CDash to support a 4th test result type?

CDash considers a timeout as a failure. The database backend will 
definitely allow for adding a new category. The main issue is the space 
on the main dashboard, which is pretty tricky to accommodate. We are 
always open to suggestions.

> 5. Similarly, MTT can also accept various performance data.  We have some (crude) graphing functionality in the MTT web reporter to show some performance metrics.  Is it possible to extend CDash to include more than just correctness data?

There is actually a feature request for this in the bug tracker. This is 
something we'd like to see implemented.

> 6. I see that you can add notes to CDash results -- are those notes searchable?  E.g., if I add "This problem fixed in SVN r12345" to some results, can I search on "r12345" to find it?

The notes are stored as text in the database so they would be 
searchable. We just never had the need for it, but it would be a great 
feature.

> 7. Is there a public roadmap of future / planned CDash features?

We usually prioritize tasks before each development cycles based on the 
bug/features stored in the bug tracker. We are usually pushing features 
based on the need of the different projects using CDash so it usually 
moves a little bit.

> 8. I see that CDash has a "daily dashboard" philosophy.  Is it possible to refresh that dashboard more than once a day?  We actually have a 24 hour testing cycle; it would be nice to be able to update the daily dashboard every so often (maybe every hour, or every 4 hours, or somesuch).

The pages are actually dynamic. There is a 24h window display but the 
builds are displayed in real-time as soon as they are submitted to CDash.

> (if it's easier to discuss this stuff on a phone call, let me know -- thanks!)

I hope, this is answering most of your questions/concerns. We would be 
happy to discuss with you over the phone if that's easier.

Julien