[Cdash] Newbie questions
julien.jomier at kitware.com
Fri Dec 3 14:15:10 UTC 2010
Welcome to CDash! (and these are not bad questions :)).
CDash is really meant to be generic and we know some users are not using
CMake/CTest to do the submission of the project. We actually have used
CDash to report testing from PHP applications and we use it on a regular
basis to test our digital archiving system (MIDAS). So I think it's
definitely doable, as long as we can translate the report to an XML
format that CDash would understand. We are also open to add new features
that would be missing from CDash.
> Is the submission XML schema documented anywhere? Are there any guarantees about the stability of this XML scheme between versions of CDash?
The documentation regarding the schema is located here:
It is more a validation schema than a full documentation but it should
get you started. Don't hesitate if you have any questions. We cannot
guarantee that the XML schema won't change, because we add new tags when
we need it in order to improve the system. That said, we rarely remove
tags (we actually never did). The base XML schema hasn't change in the
last seven years or so (taken from Dart, the predecessor of CDash).
> 1. We test distribution tarballs, not SVN checkouts. Is that against the philosophy of CDash?
Definitely not, any kind of testing is welcome!
> 2. A major theme of our web reporter is aggregation of data, and a drill-down philosophy to find the precise data that you're looking for. For example, on a good night, we might have *many* of what CDash calls "Configure" executions. We like to group these together by various attributes (e.g., tarball version, organization, architecture, compiler suite, configure option set, ...etc.) and then allow a web user to drill down to the "Configure" executions that they want to see/examine.
> You can see this in the URL that I cited above; every item in the results table is clickable. Clicking on a result table narrows the query to similar results. You keep drilling down until you find just the result set that you want, and then you click on the "Details" button to see the gory details.
> Is aggregation like this possible in CDash?
This is interesting. CDash has this notion of "tracks/groups" which
allows to group "builds" together so that would be an option (maybe not
ideal though). The other option would be to come up with a new view of
the data with a grouping concept. This is something we would have to
> 3. What is the philosophy of contribution to CDash? I see that it's open source and the SVN is available, but all of the contributors are listed as being at Kitware (http://www.cdash.org/cdash/project/parti.html). Are you open to code contributions from elsewhere? (i.e., if we buy into CDash, we may want to submit stuff upstream)
We are always welcoming contributions from other people. We have
actually integrated many feature requests submitted by developers via
the bug tracker. We have to be careful with the contributions to CDash
(as well as CMake/CTest) because it's affecting a lot of projects. We
are usually quick at integrating patches and bug fixes submitted to us
after complete review. We should probably upgrade this page with the
authors of the patches (thanks for pointing this out).
> 4. CDash seems to define 3 test outcoming: Pass, Fail, NotRun (MTT calls this "Skip"). MTT also has a "Timeout" result -- meaning that the test didn't meet the criteria for passing or failing; it was still running when a timeout expired (so the MTT client killed it and moved on to the next test). Is it possible to extend CDash to support a 4th test result type?
CDash considers a timeout as a failure. The database backend will
definitely allow for adding a new category. The main issue is the space
on the main dashboard, which is pretty tricky to accommodate. We are
always open to suggestions.
> 5. Similarly, MTT can also accept various performance data. We have some (crude) graphing functionality in the MTT web reporter to show some performance metrics. Is it possible to extend CDash to include more than just correctness data?
There is actually a feature request for this in the bug tracker. This is
something we'd like to see implemented.
> 6. I see that you can add notes to CDash results -- are those notes searchable? E.g., if I add "This problem fixed in SVN r12345" to some results, can I search on "r12345" to find it?
The notes are stored as text in the database so they would be
searchable. We just never had the need for it, but it would be a great
> 7. Is there a public roadmap of future / planned CDash features?
We usually prioritize tasks before each development cycles based on the
bug/features stored in the bug tracker. We are usually pushing features
based on the need of the different projects using CDash so it usually
moves a little bit.
> 8. I see that CDash has a "daily dashboard" philosophy. Is it possible to refresh that dashboard more than once a day? We actually have a 24 hour testing cycle; it would be nice to be able to update the daily dashboard every so often (maybe every hour, or every 4 hours, or somesuch).
The pages are actually dynamic. There is a 24h window display but the
builds are displayed in real-time as soon as they are submitted to CDash.
> (if it's easier to discuss this stuff on a phone call, let me know -- thanks!)
I hope, this is answering most of your questions/concerns. We would be
happy to discuss with you over the phone if that's easier.
More information about the CDash