[Cdash] Newbie questions

[Jeff Squyres]

Greetings all; I have trawled the CDash pages and have a few "noob" kinds of questions.  Apologies for the length of this mail!

Short version:
==============

We have a custom distributed testing client that we're quite happy with (i.e., our build system does not use CMake and we don't use CTest), but our web reporter kinda sucks.  CDash looks *much* better than ours.  I see in http://www.cmake.org/Wiki/CDash:FAQ#I.27m_not_using_CMake.2FCTest.2C_can_I_still_use_CDash.3F:

-----
I'm not using CMake/CTest, can I still use CDash?

Yes, CDash expects the XML files to be sent via HTTP PUT...
-----

Is the submission XML schema documented anywhere?  Are there any guarantees about the stability of this XML scheme between versions of CDash?

Many thanks for your time.

More details:
=============

I'm one of the developers of the Open MPI project (www.open-mpi.org).  We have a large code base that is tested by various organizations around the world on a nightly basis.  Several years ago, we researched distributed testing tools and concluded that none really did what we wanted, so we wrote our own: the MPI Testing Tool (MTT).  It ended up not being specific to MPI -- we really could have called it the "Middleware Testing Tool", for example.

Our MTT client has proved to be quite versatile and useful to us -- it has a somewhat-steep learning curve, but once you get over that, it offers a huge amount of flexibility that we need in testing the many, many different code paths through our code base.

Specifically, we have many, many Autoconf ./configure command line options that enable/disable parts of our code base, and we support many different compiler suites and operating systems (e.g., lots of different compilers on Linux).  Additionally, we have a few short of 7 billion run-time configuration options (ok, I'm exaggerating -- but there are literally hundreds of them).  Our MTT client is able to automate testing on huge sets of compile-time and run-time options so that we can test many, many different code paths.  We aggregate all this data in our MTT web reporter.

In the past 24 hours, for example, I see the following via our MTT web reporter:

- 74 equivalents to CDash's "Configure" phase
- 501 equivalents to CDash's "Build" phase
- 286K+ equivalents of CDash's "Run" phase

You can see the data here: (beware: it's somewhat slow to load :-\ )

    http://www.open-mpi.org/mtt/index.php?do_redir=1927

Although we like the MTT client a lot, the web reporter+database pair is quite slow and lacking many features; it represents a significant weak point in MTT.  There's a few features that CDash doesn't (seem to) do that our web reporter has, but CDash seems to be superior in just about every other way.  CDash therefore looks quite intriguing to us.

We were wondering if we could have our MTT client simply submit its results to CDash.  Indeed, the result phases of MTT seem to be pretty close to CDash's:

  MTT phase name   CDash phase name
- MPI Install   -> Configure
- Test build    -> Build
- Test run      -> Test

The above-referenced FAQ gives me great hope that this is actually possible.  I'm sure there will need to be some level of translation between the data that the MTT client gathers and what CDash expects, but it *seems* like it could be possible.

Any information on the XML submission schema would be appreciated, as well as any thoughts you might have on the feasibility of the idea of using our client with CDash.

Here's a few other questions that I have (if you've made it down this far!):

1. We test distribution tarballs, not SVN checkouts.  Is that against the philosophy of CDash?

2. A major theme of our web reporter is aggregation of data, and a drill-down philosophy to find the precise data that you're looking for.  For example, on a good night, we might have *many* of what CDash calls "Configure" executions.  We like to group these together by various attributes (e.g., tarball version, organization, architecture, compiler suite, configure option set, ...etc.) and then allow a web user to drill down to the "Configure" executions that they want to see/examine.  

You can see this in the URL that I cited above; every item in the results table is clickable.  Clicking on a result table narrows the query to similar results.  You keep drilling down until you find just the result set that you want, and then you click on the "Details" button to see the gory details.

Is aggregation like this possible in CDash?

3. What is the philosophy of contribution to CDash?  I see that it's open source and the SVN is available, but all of the contributors are listed as being at Kitware (http://www.cdash.org/cdash/project/parti.html).  Are you open to code contributions from elsewhere?  (i.e., if we buy into CDash, we may want to submit stuff upstream)

4. CDash seems to define 3 test outcoming: Pass, Fail, NotRun (MTT calls this "Skip").  MTT also has a "Timeout" result -- meaning that the test didn't meet the criteria for passing or failing; it was still running when a timeout expired (so the MTT client killed it and moved on to the next test).  Is it possible to extend CDash to support a 4th test result type?

5. Similarly, MTT can also accept various performance data.  We have some (crude) graphing functionality in the MTT web reporter to show some performance metrics.  Is it possible to extend CDash to include more than just correctness data?

6. I see that you can add notes to CDash results -- are those notes searchable?  E.g., if I add "This problem fixed in SVN r12345" to some results, can I search on "r12345" to find it?

7. Is there a public roadmap of future / planned CDash features?

8. I see that CDash has a "daily dashboard" philosophy.  Is it possible to refresh that dashboard more than once a day?  We actually have a 24 hour testing cycle; it would be nice to be able to update the daily dashboard every so often (maybe every hour, or every 4 hours, or somesuch).

That's probably enough questions for now.  Many thanks for your time!

(if it's easier to discuss this stuff on a phone call, let me know -- thanks!)

-- 
Jeff Squyres
jsquyres at cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/