[vtk-developers] VTK test code coverage

[David Cole]

I have always thought it should be a test failure (by default) if the
output contains the error string from encountering a vtkError while
running the test. Any exceptions to that rule would be simply to test
the error mechanism itself.

I know there are some tests that do produce Error messages when I run
them on my machine...

Seems like those should be test failures, but some are not.


On Wed, Sep 5, 2012 at 2:46 PM, Yuanxin Liu <leo.liu at kitware.com> wrote:
> Hi, Bill,
>   A separate but related question:  What should be the default behavior of
> tests regarding "ERROR" strings in the standard output?  Currently, unless
> you explicitly do something like set_tests_properties(.... PROPERTIES
> FAIL_REGULAR_EXPRESSION "Error") the tests will pass.  This seems a little
> dangerous to me.
>
> Leo
>
> On Tue, Sep 4, 2012 at 2:58 PM, Bill Lorensen <bill.lorensen at gmail.com>
> wrote:
>>
>> Folks,
>>
>> VTK's test code coverage is not very good. Currently it stands at
>> about 62% of the code. I think the pre/post modularization coverage
>> rates are similar.
>>
>> One of the lessons we learned from ITK's modularization is that many
>> of the ITK tests were integration/regression tests versus unit
>> tests. Integration/regression tests tend to test multiple classes, but
>> fail to fully cover the code in individual classes.
>>
>> Integration/regression tests are certainly important because they test
>> the code as it may be used in an application.  In fact, many of VTK's
>> tests are derived from examples in the original text. Their purpose was to
>> make sure the code actually did what the text said it would do.
>>
>> But, having 40% of VTK's code untested is alarming.
>>
>> Last year, I picked valgrind defects as my Fall Project:
>> http://itk.org/Wiki/VTK/SoftwareQuality/Valgrind/Fall2011
>>
>> I have adapted some ITK scripts to easily evaluate code coverage for VTK.
>> There is a script in
>> Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh that
>> facilitates generating code coverage for a set of unit tests.
>>
>>
>> For example, running this command from your build directory
>> (instrumented for coverage)
>>
>> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
>> Domains/Chemistry -R vtkDomainsChemistry
>> shows that the Domains/Chemistry module has 67.6% code coverage
>> while
>> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
>> Geovis/Core -R vtkGeovisCore
>> shows that the Geovis/Core modules has 11.2% code coverage/
>>
>> This year's Fall Project is code coverage. This is a bit ironic,
>> because for my GE Six Sigma Project in 1995, I chose code
>> coverage.
>>
>> What can you do to help?
>> If you are adding or modifying new classes to vtk, please try to
>> add/enhance the tests for your class.
>> I'll be adding gerrit topics for tests this fall. Please help me push them
>> through the review process
>>
>>
>>
>> Also, I'll be rejecting gerrit topics that add new classes without good
>> unit tests.
>>
>> Bill
>> '
>>
>>
>>
>> --
>> Unpaid intern in BillsBasement at noware dot com
>>
>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.vtk.org/mailman/listinfo/vtk-developers
>>
>>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtk-developers
>
>