[vtk-developers] [Cdash] CDash query for all failing tests

[David E DeMarle]

A completed option space inspector script (vtk_submitter_summary.py)
is going through gerrit review. See:
http://review.source.kitware.com/#/c/9801/

It would be much more convenient if cdash would gather and present the
information that that script gathers on demand.

I'll can help implement in cdash if we get time to take it up.

thanks,

David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909


On Tue, Feb 12, 2013 at 9:05 PM, David E DeMarle
<dave.demarle at kitware.com> wrote:
> The second script does essentially the same thing that Dave Cole did years ago
> Ie:
> http://www.cmake.org/Wiki/CMake_Useful_Variables/Get_Variables_From_CMake_Dashboards
> I assume that was in the dart days as it didn't work for me right off
> the bat, so I decided to roll my own.
>
> David E DeMarle
> Kitware, Inc.
> R&D Engineer
> 21 Corporate Drive
> Clifton Park, NY 12065-8662
> Phone: 518-881-4909
>
>
> On Tue, Feb 12, 2013 at 9:02 PM, David E DeMarle
> <dave.demarle at kitware.com> wrote:
>> If you run vtk_fail_summary.py it will summarize all the test failures
>> on the nightly expected group, from there you have to dig deeper than
>> the cdash web api will let you go.
>>
>> To go down that route, take a look at vtk_submitter_summary.py. It
>> scrapes the raw html pages to get the results of a particular
>> test on every machine. The script and the accompanying test (once I finish and
>> commit them) will give us the details of each submitter's configuration, so
>> that we can inspect the testing space and know what configurations we
>> are testing and what we aren't.
>>
>> David E DeMarle
>> Kitware, Inc.
>> R&D Engineer
>> 21 Corporate Drive
>> Clifton Park, NY 12065-8662
>> Phone: 518-881-4909
>>
>>
>> On Tue, Feb 12, 2013 at 4:04 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>> Bill,
>>>
>>> Thanks. I knew how to do that. I was hoping I could see all the
>>> failing test details. I'll have to do it by hand. Or maybe I can
>>> figure out a way to scrape the cdash pages.
>>>
>>> Currently VTK has 18 Nightly Expected builds with test failures
>>> (actually that is every build in that group). I'd like to generate a
>>> page (or spreadsheet) that shows every failing test (today there are
>>> 123 total tests failing).
>>>
>>> Bill
>>>
>>> On Tue, Feb 12, 2013 at 3:23 PM, Bill Hoffman <bill.hoffman at kitware.com> wrote:
>>>> On 2/12/2013 1:53 PM, Bill Lorensen wrote:
>>>>>
>>>>> Folks,
>>>>>
>>>>> Is there a cdash query that can provide, on a single page, links to
>>>>> all failing tests for a given group, say "Nightly Expected"?
>>>>>
>>>>> Bill
>>>>> _______________________________________________
>>>>> Powered by www.kitware.com
>>>>>
>>>>> Visit other Kitware open-source projects at
>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>
>>>>> Follow this link to subscribe/unsubscribe:
>>>>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>>>>
>>>>>
>>>>
>>>> You can do something like this:
>>>>
>>>> http://open.cdash.org/index.php?project=CMake&date=2013-02-12&filtercount=2&showfilters=1&filtercombine=and&field1=groupname/string&compare1=63&value1=Nightly%20Expected&field2=testsfailed/number&compare2=43&value2=0
>>>>
>>>> If you create a filter you can match the group name.  Unfortunately, you can
>>>> not show errors and test failures.  You can only match all or any of the
>>>> filters.   I think CDash needs to be able to have the any or all be applied
>>>> to each of the filters so you can create a filter that will show you any red
>>>> in a group.  But you can show all failing tests for a group.  :)
>>>>
>>>> -Bill
>>>>
>>>>
>>>> --
>>>> Bill Hoffman
>>>> Kitware, Inc.
>>>> 28 Corporate Drive
>>>> Clifton Park, NY 12065
>>>> bill.hoffman at kitware.com
>>>> http://www.kitware.com
>>>> 518 881-4905 (Direct)
>>>> 518 371-3971 x105
>>>> Fax (518) 371-4573
>>>
>>>
>>>
>>> --
>>> Unpaid intern in BillsBasement at noware dot com
>>> _______________________________________________
>>> Cdash mailing list
>>> Cdash at public.kitware.com
>>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash