[vtk-developers] VTK test code coverage

[Bill Lorensen]

Coverage has been low for a long time, prior to VTK6.


On Thu, Sep 6, 2012 at 12:50 PM, Yuanxin Liu <leo.liu at kitware.com> wrote:

> Another reason for the low testing coverage might be that some tests are
> no longer build after VTK modularization.  A common pattern I saw was a
> testing folder with no CMakeLists.txt in it or does not include all test
> files, but I think things are look better and better, though.
>
> Leo
>
> On Wed, Sep 5, 2012 at 11:19 PM, Bill Lorensen <bill.lorensen at gmail.com>wrote:
>
>> I agree with Dave that unless explicitly testing an error condition, it
>> should be flagged an error.
>>
>> To explicitly check for an error you can do something like I did in
>> Imaging/Hybrid/Testing/Cxx/TestSampleFunction.cxx
>>
>> With ITK it is a bit easier because ITK throws exceptions that can be
>> caught by the test program.
>>
>> Bill
>>
>> On Wed, Sep 5, 2012 at 3:47 PM, David Cole <david.cole at kitware.com>wrote:
>>
>>> I have always thought it should be a test failure (by default) if the
>>> output contains the error string from encountering a vtkError while
>>> running the test. Any exceptions to that rule would be simply to test
>>> the error mechanism itself.
>>>
>>> I know there are some tests that do produce Error messages when I run
>>> them on my machine...
>>>
>>> Seems like those should be test failures, but some are not.
>>>
>>>
>>> On Wed, Sep 5, 2012 at 2:46 PM, Yuanxin Liu <leo.liu at kitware.com> wrote:
>>> > Hi, Bill,
>>> >   A separate but related question:  What should be the default
>>> behavior of
>>> > tests regarding "ERROR" strings in the standard output?  Currently,
>>> unless
>>> > you explicitly do something like set_tests_properties(.... PROPERTIES
>>> > FAIL_REGULAR_EXPRESSION "Error") the tests will pass.  This seems a
>>> little
>>> > dangerous to me.
>>> >
>>> > Leo
>>> >
>>> > On Tue, Sep 4, 2012 at 2:58 PM, Bill Lorensen <bill.lorensen at gmail.com
>>> >
>>> > wrote:
>>> >>
>>> >> Folks,
>>> >>
>>> >> VTK's test code coverage is not very good. Currently it stands at
>>> >> about 62% of the code. I think the pre/post modularization coverage
>>> >> rates are similar.
>>> >>
>>> >> One of the lessons we learned from ITK's modularization is that many
>>> >> of the ITK tests were integration/regression tests versus unit
>>> >> tests. Integration/regression tests tend to test multiple classes, but
>>> >> fail to fully cover the code in individual classes.
>>> >>
>>> >> Integration/regression tests are certainly important because they test
>>> >> the code as it may be used in an application.  In fact, many of VTK's
>>> >> tests are derived from examples in the original text. Their purpose
>>> was to
>>> >> make sure the code actually did what the text said it would do.
>>> >>
>>> >> But, having 40% of VTK's code untested is alarming.
>>> >>
>>> >> Last year, I picked valgrind defects as my Fall Project:
>>> >> http://itk.org/Wiki/VTK/SoftwareQuality/Valgrind/Fall2011
>>> >>
>>> >> I have adapted some ITK scripts to easily evaluate code coverage for
>>> VTK.
>>> >> There is a script in
>>> >> Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh that
>>> >> facilitates generating code coverage for a set of unit tests.
>>> >>
>>> >>
>>> >> For example, running this command from your build directory
>>> >> (instrumented for coverage)
>>> >>
>>> >> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
>>> >> Domains/Chemistry -R vtkDomainsChemistry
>>> >> shows that the Domains/Chemistry module has 67.6% code coverage
>>> >> while
>>> >> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
>>> >> Geovis/Core -R vtkGeovisCore
>>> >> shows that the Geovis/Core modules has 11.2% code coverage/
>>> >>
>>> >> This year's Fall Project is code coverage. This is a bit ironic,
>>> >> because for my GE Six Sigma Project in 1995, I chose code
>>> >> coverage.
>>> >>
>>> >> What can you do to help?
>>> >> If you are adding or modifying new classes to vtk, please try to
>>> >> add/enhance the tests for your class.
>>> >> I'll be adding gerrit topics for tests this fall. Please help me push
>>> them
>>> >> through the review process
>>> >>
>>> >>
>>> >>
>>> >> Also, I'll be rejecting gerrit topics that add new classes without
>>> good
>>> >> unit tests.
>>> >>
>>> >> Bill
>>> >> '
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Unpaid intern in BillsBasement at noware dot com
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> Powered by www.kitware.com
>>> >>
>>> >> Visit other Kitware open-source projects at
>>> >> http://www.kitware.com/opensource/opensource.html
>>> >>
>>> >> Follow this link to subscribe/unsubscribe:
>>> >> http://www.vtk.org/mailman/listinfo/vtk-developers
>>> >>
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > Powered by www.kitware.com
>>> >
>>> > Visit other Kitware open-source projects at
>>> > http://www.kitware.com/opensource/opensource.html
>>> >
>>> > Follow this link to subscribe/unsubscribe:
>>> > http://www.vtk.org/mailman/listinfo/vtk-developers
>>> >
>>> >
>>>
>>
>>
>>
>> --
>> Unpaid intern in BillsBasement at noware dot com
>>
>>
>


-- 
Unpaid intern in BillsBasement at noware dot com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20120906/29951de0/attachment.html>