[Cdash] Newbie questions
neundorf at kde.org
Sun Dec 5 14:07:20 UTC 2010
On Friday 03 December 2010, Julien Jomier wrote:
> Hi Jeff,
> Welcome to CDash! (and these are not bad questions :)).
> CDash is really meant to be generic and we know some users are not using
> CMake/CTest to do the submission of the project. We actually have used
> CDash to report testing from PHP applications and we use it on a regular
> basis to test our digital archiving system (MIDAS). So I think it's
> definitely doable, as long as we can translate the report to an XML
> format that CDash would understand. We are also open to add new features
> that would be missing from CDash.
> > Is the submission XML schema documented anywhere? Are there any
> > guarantees about the stability of this XML scheme between versions of
> > CDash?
> The documentation regarding the schema is located here:
> It is more a validation schema than a full documentation but it should
> get you started. Don't hesitate if you have any questions. We cannot
> guarantee that the XML schema won't change, because we add new tags when
> we need it in order to improve the system. That said, we rarely remove
> tags (we actually never did). The base XML schema hasn't change in the
> last seven years or so (taken from Dart, the predecessor of CDash).
> > 1. We test distribution tarballs, not SVN checkouts. Is that against the
> > philosophy of CDash?
> Definitely not, any kind of testing is welcome!
> > 2. A major theme of our web reporter is aggregation of data, and a
> > drill-down philosophy to find the precise data that you're looking for.
> > For example, on a good night, we might have *many* of what CDash calls
> > "Configure" executions. We like to group these together by various
> > attributes (e.g., tarball version, organization, architecture, compiler
> > suite, configure option set, ...etc.) and then allow a web user to drill
> > down to the "Configure" executions that they want to see/examine. You can
> > see this in the URL that I cited above; every item in the results table
> > is clickable. Clicking on a result table narrows the query to similar
> > results. You keep drilling down until you find just the result set that
> > you want, and then you click on the "Details" button to see the gory
> > details. Is aggregation like this possible in CDash?
> This is interesting. CDash has this notion of "tracks/groups" which
> allows to group "builds" together so that would be an option (maybe not
> ideal though). The other option would be to come up with a new view of
> the data with a grouping concept. This is something we would have to
> > 3. What is the philosophy of contribution to CDash? I see that it's open
> > source and the SVN is available, but all of the contributors are listed
> > as being at Kitware (http://www.cdash.org/cdash/project/parti.html). Are
> > you open to code contributions from elsewhere? (i.e., if we buy into
> > CDash, we may want to submit stuff upstream)
> We are always welcoming contributions from other people. We have
> actually integrated many feature requests submitted by developers via
> the bug tracker. We have to be careful with the contributions to CDash
> (as well as CMake/CTest) because it's affecting a lot of projects. We
> are usually quick at integrating patches and bug fixes submitted to us
> after complete review. We should probably upgrade this page with the
> authors of the patches (thanks for pointing this out).
> > 4. CDash seems to define 3 test outcoming: Pass, Fail, NotRun (MTT calls
> > this "Skip"). MTT also has a "Timeout" result -- meaning that the test
> > didn't meet the criteria for passing or failing; it was still running
> > when a timeout expired (so the MTT client killed it and moved on to the
> > next test). Is it possible to extend CDash to support a 4th test result
> > type?
> CDash considers a timeout as a failure. The database backend will
> definitely allow for adding a new category. The main issue is the space
> on the main dashboard, which is pretty tricky to accommodate. We are
> always open to suggestions.
> > 5. Similarly, MTT can also accept various performance data. We have some
> > (crude) graphing functionality in the MTT web reporter to show some
> > performance metrics. Is it possible to extend CDash to include more than
> > just correctness data?
> There is actually a feature request for this in the bug tracker. This is
> something we'd like to see implemented.
E.g. this one:
More information about the CDash