[Cdash] Newbie questions

[Alexander Neundorf]

On Friday 03 December 2010, Julien Jomier wrote:
> Hi Jeff,
>
> Welcome to CDash! (and these are not bad questions :)).
>
> CDash is really meant to be generic and we know some users are not using
> CMake/CTest to do the submission of the project. We actually have used
> CDash to report testing from PHP applications and we use it on a regular
> basis to test our digital archiving system (MIDAS). So I think it's
> definitely doable, as long as we can translate the report to an XML
> format that CDash would understand. We are also open to add new features
> that would be missing from CDash.
>
> > Is the submission XML schema documented anywhere?  Are there any
> > guarantees about the stability of this XML scheme between versions of
> > CDash?
>
> The documentation regarding the schema is located here:
>
>    http://public.kitware.com/Wiki/CDash:XML
>
> It is more a validation schema than a full documentation but it should
> get you started. Don't hesitate if you have any questions. We cannot
> guarantee that the XML schema won't change, because we add new tags when
> we need it in order to improve the system. That said, we rarely remove
> tags (we actually never did). The base XML schema hasn't change in the
> last seven years or so (taken from Dart, the predecessor of CDash).
>
> > 1. We test distribution tarballs, not SVN checkouts.  Is that against the
> > philosophy of CDash?
>
> Definitely not, any kind of testing is welcome!
>
> > 2. A major theme of our web reporter is aggregation of data, and a
> > drill-down philosophy to find the precise data that you're looking for. 
> > For example, on a good night, we might have *many* of what CDash calls
> > "Configure" executions.  We like to group these together by various
> > attributes (e.g., tarball version, organization, architecture, compiler
> > suite, configure option set, ...etc.) and then allow a web user to drill
> > down to the "Configure" executions that they want to see/examine. You can
> > see this in the URL that I cited above; every item in the results table
> > is clickable.  Clicking on a result table narrows the query to similar
> > results.  You keep drilling down until you find just the result set that
> > you want, and then you click on the "Details" button to see the gory
> > details. Is aggregation like this possible in CDash?
>
> This is interesting. CDash has this notion of "tracks/groups" which
> allows to group "builds" together so that would be an option (maybe not
> ideal though). The other option would be to come up with a new view of
> the data with a grouping concept. This is something we would have to
> investigate.
>
> > 3. What is the philosophy of contribution to CDash?  I see that it's open
> > source and the SVN is available, but all of the contributors are listed
> > as being at Kitware (http://www.cdash.org/cdash/project/parti.html).  Are
> > you open to code contributions from elsewhere?  (i.e., if we buy into
> > CDash, we may want to submit stuff upstream)
>
> We are always welcoming contributions from other people. We have
> actually integrated many feature requests submitted by developers via
> the bug tracker. We have to be careful with the contributions to CDash
> (as well as CMake/CTest) because it's affecting a lot of projects. We
> are usually quick at integrating patches and bug fixes submitted to us
> after complete review. We should probably upgrade this page with the
> authors of the patches (thanks for pointing this out).
>
> > 4. CDash seems to define 3 test outcoming: Pass, Fail, NotRun (MTT calls
> > this "Skip").  MTT also has a "Timeout" result -- meaning that the test
> > didn't meet the criteria for passing or failing; it was still running
> > when a timeout expired (so the MTT client killed it and moved on to the
> > next test).  Is it possible to extend CDash to support a 4th test result
> > type?
>
> CDash considers a timeout as a failure. The database backend will
> definitely allow for adding a new category. The main issue is the space
> on the main dashboard, which is pretty tricky to accommodate. We are
> always open to suggestions.
>
> > 5. Similarly, MTT can also accept various performance data.  We have some
> > (crude) graphing functionality in the MTT web reporter to show some
> > performance metrics.  Is it possible to extend CDash to include more than
> > just correctness data?
>
> There is actually a feature request for this in the bug tracker. This is
> something we'd like to see implemented.

E.g. this one:
http://public.kitware.com/Bug/view.php?id=10454

Alex