[vtk-developers] [Cdash] CDash query for all failing tests

[David Cole]

Don't forget, there are also these pages already available:

This one [ http://open.cdash.org/testOverview.php?project=VTK ] shows a summary of the names of the tests that are failing in a given group. After you get to the page, you have to choose "Nightly Expected" to see those test failures. This is the closest thing there is to "a summary page that shows what's wrong with a given group", but there are no details on that page, and you're still 2 clicks away from seeing details.

This one [ http://open.cdash.org/queryTests.php?project=VTK&date=&limit=10 ] allows you to choose filter parameters, but (unfortunately) there is no easy way on that page to limit the output to a given build group. And again, you're at least 1 click away from seeing any details.

It would be better to modify CDash to provide the desired page than it would to introduce scraping scripts. If the scraping scripts run "too much" or automatically, rather than strictly on demand by developers when they need it, it will just put a mostly unnecessary burden on the CDash server. This page should only be produced *on demand*, not on an automatic schedule.


D


 

 

-----Original Message-----
From: David E DeMarle <dave.demarle at kitware.com>
To: Bill Lorensen <bill.lorensen at gmail.com>
Cc: VTK Developers <vtk-developers at vtk.org>; cdash <cdash at public.kitware.com>; Bill Hoffman <bill.hoffman at kitware.com>
Sent: Tue, Feb 12, 2013 9:06 pm
Subject: Re: [vtk-developers] [Cdash]  CDash query for all failing tests


The second script does essentially the same thing that Dave Cole did years ago
Ie:
http://www.cmake.org/Wiki/CMake_Useful_Variables/Get_Variables_From_CMake_Dashboards
I assume that was in the dart days as it didn't work for me right off
the bat, so I decided to roll my own.

David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909


On Tue, Feb 12, 2013 at 9:02 PM, David E DeMarle
<dave.demarle at kitware.com> wrote:
> If you run vtk_fail_summary.py it will summarize all the test failures
> on the nightly expected group, from there you have to dig deeper than
> the cdash web api will let you go.
>
> To go down that route, take a look at vtk_submitter_summary.py. It
> scrapes the raw html pages to get the results of a particular
> test on every machine. The script and the accompanying test (once I finish and
> commit them) will give us the details of each submitter's configuration, so
> that we can inspect the testing space and know what configurations we
> are testing and what we aren't.
>
> David E DeMarle
> Kitware, Inc.
> R&D Engineer
> 21 Corporate Drive
> Clifton Park, NY 12065-8662
> Phone: 518-881-4909
>
>
> On Tue, Feb 12, 2013 at 4:04 PM, Bill Lorensen <bill.lorensen at gmail.com> 
wrote:
>> Bill,
>>
>> Thanks. I knew how to do that. I was hoping I could see all the
>> failing test details. I'll have to do it by hand. Or maybe I can
>> figure out a way to scrape the cdash pages.
>>
>> Currently VTK has 18 Nightly Expected builds with test failures
>> (actually that is every build in that group). I'd like to generate a
>> page (or spreadsheet) that shows every failing test (today there are
>> 123 total tests failing).
>>
>> Bill
>>
>> On Tue, Feb 12, 2013 at 3:23 PM, Bill Hoffman <bill.hoffman at kitware.com> 
wrote:
>>> On 2/12/2013 1:53 PM, Bill Lorensen wrote:
>>>>
>>>> Folks,
>>>>
>>>> Is there a cdash query that can provide, on a single page, links to
>>>> all failing tests for a given group, say "Nightly Expected"?
>>>>
>>>> Bill
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>>>
>>>>
>>>
>>> You can do something like this:
>>>
>>> http://open.cdash.org/index.php?project=CMake&date=2013-02-12&filtercount=2&showfilters=1&filtercombine=and&field1=groupname/string&compare1=63&value1=Nightly%20Expected&field2=testsfailed/number&compare2=43&value2=0
>>>
>>> If you create a filter you can match the group name.  Unfortunately, you can
>>> not show errors and test failures.  You can only match all or any of the
>>> filters.   I think CDash needs to be able to have the any or all be applied
>>> to each of the filters so you can create a filter that will show you any red
>>> in a group.  But you can show all failing tests for a group.  :)
>>>
>>> -Bill
>>>
>>>
>>> --
>>> Bill Hoffman
>>> Kitware, Inc.
>>> 28 Corporate Drive
>>> Clifton Park, NY 12065
>>> bill.hoffman at kitware.com
>>> http://www.kitware.com
>>> 518 881-4905 (Direct)
>>> 518 371-3971 x105
>>> Fax (518) 371-4573
>>
>>
>>
>> --
>> Unpaid intern in BillsBasement at noware dot com
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html

Follow this link to subscribe/unsubscribe:
http://www.vtk.org/mailman/listinfo/vtk-developers


 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20130213/a104f297/attachment.html>