[Cdash] cdash API

[David E DeMarle]

On Wed, Oct 3, 2012 at 10:48 AM, David Cole <david.cole at kitware.com> wrote:
> On Wed, Oct 3, 2012 at 9:59 AM, David E DeMarle
> <dave.demarle at kitware.com> wrote:
>> On Wed, Oct 3, 2012 at 9:35 AM, David E DeMarle
>> <dave.demarle at kitware.com> wrote:
>>> I'm looking for more information on cdash's external API.
>>> See:
>>> http://public.kitware.com/Bug/view.php?id=8225
>>>
>>> Is there more to the documentation of the API than this?
>>> http://public.kitware.com/Wiki/CDash:API
>>>
>>> I am hoping to write some python scripts that produce additional
>>> reports that you can't easily get off a given project's website and
>>> thus take a lot of manual effort to build up.
>>>
>>> Specifically:
>>> * for a given project, day, and section, produce a table that shows
>>> every test that fails and the lists of machines that fail it (i.e.
>>> test -> submission instead of submission -> test)
>>
>> http://open.cdash.org/api/?method=build&task=sitetestfailures&project=VTK&group=Nightly%20Expected
>> Seems like what I am looking for in this case.
>>
>>> * for a given project, day, and section, produce a table that shows
>>> every compilation warning and the list of machines that produce it
>>> * for a given project, day, and section, produce a table that shows
>>> all submitters configurations (what is the platform and option space
>>> coverage)
>>> * across many projects, show the submitters and the start and stop
>>> times of their submissions
>>>
>>> Any information that shows how to go about doing that will be appreciated.
>>>
>>> thanks,
>>>
>>> David E DeMarle
>>> Kitware, Inc.
>>> R&D Engineer
>>> 21 Corporate Drive
>>> Clifton Park, NY 12065-8662
>>> Phone: 518-881-4909
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>
>
> There is also this page, where you can click "Show Filters" and search
> on many different fields. It would be nice to add "group" as a
> possible query parameter here:
>
> http://open.cdash.org/queryTests.php?project=VTK&date=&limit=200&filtercount=1&showfilters=1&limit=200&field1=testname/string&compare1=63&value1=
>
> You can get to this page initially by clicking on the "Dashboard" drop
> down menu, and choosing "Tests Query."
>
>
> And there is also this page, which does have "group" selection
> capability, but lists all failing tests alphabetically:
>
> http://open.cdash.org/testOverview.php?project=VTK&date=
>
> Found under "Dashboard" > "Tests"
>

Thanks. I will check them out.

I want to avoid any "clicking". If it is manual, it takes too much
intervention and is subject to errors and laziness.

>
> Have you considered contributing to CDash and figuring out how to add
> a page (or modify an existing page) rather than just having some
> python scripts on the side? I'm sure many, many others would
> appreciate and benefit from adding this type of thing directly to
> CDash.

I have, but it will be too slow. Both for me to get up to speed and
for the changes to get integrated back in and usable on the project I
need them for a couple of weeks ago.

Here is a python filter which produces the report that maps test
failures to submitters:

import urllib
url = 'http://open.cdash.org/api/?method=build&task=sitetestfailures&project=VTK&group=Nightly%20Expected'
page = urllib.urlopen(url)
data = page.readlines()
submissions = eval(data[0])
tfails = dict()
print "-"*20, "ANALYZING", "-"*20
for skey in submissions.keys():
  submission = submissions[skey]
  bname = submission['buildname']
  bfails = submission['tests']
  if len(bfails) > 100:
    print "WARNING IGNORING ", bname, len(bfails)
    continue
  print bname, len(bfails)
  for tnum in range(0, len(bfails)):
    test = bfails[tnum]
    tname = test['name']
    if not tname in tfails:
       tfails[tname] = list()
    tfails[tname].append(bname)

print "-"*20, "REPORT", "-"*20
print len(tfails)," FAILURES"
failcounts = map(lambda x: (x,len(tfails[x])), tfails.keys())
sortedfails = sorted(failcounts, key=lambda fail: fail[1])
for test in sortedfails:
  tname = test[0]
  print tname, len(tfails[tname]), tfails[tname]

>
>
> HTH,
> David C.