[vtk-developers] cdash API and VTK's most wanted list

[David E DeMarle]

Thanks for the slides they are quite informative.

I'm hoping to find time to use cdash's web API to scrape out a testing
space coverage view.

That is I want to see what OS's, compilers and graphics drivers do we
test on, as well as what wrapped languages, dependent libraries, and
other cmake options do we exercise on them.

>From a view of the test option space we can see what we cover and more
importantly don't cover, and on what we cover, where we are having
problems.

Like the test view, you _can_ build up this information manually from
cdash's pages, but it takes even longer to do so because the
information is more widely scattered (at least for VTK).

David E DeMarle
Kitware, Inc.
R&D Engineer
21 Corporate Drive
Clifton Park, NY 12065-8662
Phone: 518-881-4909


On Fri, Oct 5, 2012 at 3:31 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> Dave,
>
> Very nice. Back in the day, we used to generate a table with a row for each
> test and a column for each build. A red/green box showed pass/fail. Made it
> easy to see if it was a windows/linux/sun/hp/sgi... issue.
>
> I'll look into TestXMLFIleOutputWindow which seems to be failing strangely
> on all windows platform (except cygwin).
>
> For a description of the eary origins of testing for vtk you might look at:
> http://www.cdash.org/cdash/project/download/ExtremeTestingTalk.pdf
>
>
> On Fri, Oct 5, 2012 at 2:38 PM, David E DeMarle <dave.demarle at kitware.com>
> wrote:
>>
>> Howdy y'all,
>>
>> I've been working on cleaning the VTK dashboard lately and recently
>> learned something that is really useful - that CDash's Web API is
>> really powerful.
>>
>> See: http://public.kitware.com/Wiki/CDash:API
>>
>> Using it I wrote the attached python script which scrapes last night's
>> VTK test results and presents them in a test centric view, rather than
>> cdash's native submitter centric view.
>>
>> The test centric view is more useful I think because it helps you to
>> prioritize what _code_ or project feature is worst and should be fixed
>> soonest, as opposed to what machine is performing worst. It also gives
>> you a more realistic view of the state of the project, because it
>> won't make you falsely believe, for instance, that there are at most
>> the same number of problems as the worst dashboard shows. (20 or so vs
>> 70 today)
>>
>> Also, since it is a script it takes less than a second to generate the
>> report, whereas if you build up the same list on your own manually by
>> clicking on cdash's web pages it will take you about an hour on a
>> project (and project state) like VTK today.
>>
>> If you run the script you will see that the worst tests are:
>>
>> vtkIOExportCxx-TestGL2PSExporterVolumeRaster-VerifyRasterizedPNG , 6 ,
>> ['Mac10.7.3-clang-release-x86_64-nightly_master', 'Win32-vs9-Shared',
>> 'Win64-VS10', 'Mac10.7.3-gcc-release-x86_64-nightly_master',
>> 'Win32-vs9-Static', 'Ubuntu-Valgrind']
>> vtkInteractionStyleTcl-TestInteractorStyleTerrain , 7 , [...]
>> vtkRenderingOpenGLCxx-TestGaussianBlurPass , 7 , [...]
>> vtkCommonCore-TestXMLFileOutputWindow , 7 , [...]
>> vtkRenderingOpenGLCxx-TestTranslucentLUTDepthPeelingPass , 7 , [...]
>> ! vtkRenderingVolumeTcl-gaussian , 10 , [...]
>> ! vtkIOGeometryTcl-TestXYZMol , 10 , [...]
>> vtkIOSQLCxx-SQLiteTableReadWrite , 10 , [...]
>> vtkRenderingOpenGLCxx-TestOffAxisStereo , 14 , [...]
>> ! vtkDomainsChemistryCxx-TestProteinRibbon , 19 , [...]
>>
>> Where the ones I've marked with ! have been fixed and thus should be
>> off the most listed list tonight.
>>
>> David E DeMarle
>> Kitware, Inc.
>> R&D Engineer
>> 21 Corporate Drive
>> Clifton Park, NY 12065-8662
>> Phone: 518-881-4909
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>
>>
>
>
>
> --
> Unpaid intern in BillsBasement at noware dot com
>