[Cdash] Newbie questions

[Jeff Squyres]

Thanks for your answers Julien!

I've taken them back to the team; we'll see what we can do over here.  With the holidays coming up, we may not come back to you for more info immediately (heck, it might not be until next year at this point!).  But as we get more experience with cdash, we might come back with more questions, etc.


On Dec 3, 2010, at 9:15 AM, Julien Jomier wrote:

> Hi Jeff,
> 
> Welcome to CDash! (and these are not bad questions :)).
> 
> CDash is really meant to be generic and we know some users are not using CMake/CTest to do the submission of the project. We actually have used CDash to report testing from PHP applications and we use it on a regular basis to test our digital archiving system (MIDAS). So I think it's definitely doable, as long as we can translate the report to an XML format that CDash would understand. We are also open to add new features that would be missing from CDash.
> 
>> Is the submission XML schema documented anywhere?  Are there any guarantees about the stability of this XML scheme between versions of CDash?
> 
> The documentation regarding the schema is located here:
> 
>  http://public.kitware.com/Wiki/CDash:XML
> 
> It is more a validation schema than a full documentation but it should get you started. Don't hesitate if you have any questions. We cannot guarantee that the XML schema won't change, because we add new tags when we need it in order to improve the system. That said, we rarely remove tags (we actually never did). The base XML schema hasn't change in the last seven years or so (taken from Dart, the predecessor of CDash).
> 
>> 1. We test distribution tarballs, not SVN checkouts.  Is that against the philosophy of CDash?
> 
> Definitely not, any kind of testing is welcome!
> 
>> 2. A major theme of our web reporter is aggregation of data, and a drill-down philosophy to find the precise data that you're looking for.  For example, on a good night, we might have *many* of what CDash calls "Configure" executions.  We like to group these together by various attributes (e.g., tarball version, organization, architecture, compiler suite, configure option set, ...etc.) and then allow a web user to drill down to the "Configure" executions that they want to see/examine.
>> You can see this in the URL that I cited above; every item in the results table is clickable.  Clicking on a result table narrows the query to similar results.  You keep drilling down until you find just the result set that you want, and then you click on the "Details" button to see the gory details.
>> Is aggregation like this possible in CDash?
> 
> This is interesting. CDash has this notion of "tracks/groups" which allows to group "builds" together so that would be an option (maybe not ideal though). The other option would be to come up with a new view of the data with a grouping concept. This is something we would have to investigate.
> 
>> 3. What is the philosophy of contribution to CDash?  I see that it's open source and the SVN is available, but all of the contributors are listed as being at Kitware (http://www.cdash.org/cdash/project/parti.html).  Are you open to code contributions from elsewhere?  (i.e., if we buy into CDash, we may want to submit stuff upstream)
> 
> We are always welcoming contributions from other people. We have actually integrated many feature requests submitted by developers via the bug tracker. We have to be careful with the contributions to CDash (as well as CMake/CTest) because it's affecting a lot of projects. We are usually quick at integrating patches and bug fixes submitted to us after complete review. We should probably upgrade this page with the authors of the patches (thanks for pointing this out).
> 
>> 4. CDash seems to define 3 test outcoming: Pass, Fail, NotRun (MTT calls this "Skip").  MTT also has a "Timeout" result -- meaning that the test didn't meet the criteria for passing or failing; it was still running when a timeout expired (so the MTT client killed it and moved on to the next test).  Is it possible to extend CDash to support a 4th test result type?
> 
> CDash considers a timeout as a failure. The database backend will definitely allow for adding a new category. The main issue is the space on the main dashboard, which is pretty tricky to accommodate. We are always open to suggestions.
> 
>> 5. Similarly, MTT can also accept various performance data.  We have some (crude) graphing functionality in the MTT web reporter to show some performance metrics.  Is it possible to extend CDash to include more than just correctness data?
> 
> There is actually a feature request for this in the bug tracker. This is something we'd like to see implemented.
> 
>> 6. I see that you can add notes to CDash results -- are those notes searchable?  E.g., if I add "This problem fixed in SVN r12345" to some results, can I search on "r12345" to find it?
> 
> The notes are stored as text in the database so they would be searchable. We just never had the need for it, but it would be a great feature.
> 
>> 7. Is there a public roadmap of future / planned CDash features?
> 
> We usually prioritize tasks before each development cycles based on the bug/features stored in the bug tracker. We are usually pushing features based on the need of the different projects using CDash so it usually moves a little bit.
> 
>> 8. I see that CDash has a "daily dashboard" philosophy.  Is it possible to refresh that dashboard more than once a day?  We actually have a 24 hour testing cycle; it would be nice to be able to update the daily dashboard every so often (maybe every hour, or every 4 hours, or somesuch).
> 
> The pages are actually dynamic. There is a 24h window display but the builds are displayed in real-time as soon as they are submitted to CDash.
> 
>> (if it's easier to discuss this stuff on a phone call, let me know -- thanks!)
> 
> I hope, this is answering most of your questions/concerns. We would be happy to discuss with you over the phone if that's easier.
> 
> Julien


-- 
Jeff Squyres
jsquyres at cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/