[vtk-developers] VTK test code coverage

[Yuanxin Liu]

Another reason for the low testing coverage might be that some tests are no
longer build after VTK modularization.  A common pattern I saw was a
testing folder with no CMakeLists.txt in it or does not include all test
files, but I think things are look better and better, though.

Leo

On Wed, Sep 5, 2012 at 11:19 PM, Bill Lorensen <bill.lorensen at gmail.com>wrote:

> I agree with Dave that unless explicitly testing an error condition, it
> should be flagged an error.
>
> To explicitly check for an error you can do something like I did in
> Imaging/Hybrid/Testing/Cxx/TestSampleFunction.cxx
>
> With ITK it is a bit easier because ITK throws exceptions that can be
> caught by the test program.
>
> Bill
>
> On Wed, Sep 5, 2012 at 3:47 PM, David Cole <david.cole at kitware.com> wrote:
>
>> I have always thought it should be a test failure (by default) if the
>> output contains the error string from encountering a vtkError while
>> running the test. Any exceptions to that rule would be simply to test
>> the error mechanism itself.
>>
>> I know there are some tests that do produce Error messages when I run
>> them on my machine...
>>
>> Seems like those should be test failures, but some are not.
>>
>>
>> On Wed, Sep 5, 2012 at 2:46 PM, Yuanxin Liu <leo.liu at kitware.com> wrote:
>> > Hi, Bill,
>> >   A separate but related question:  What should be the default behavior
>> of
>> > tests regarding "ERROR" strings in the standard output?  Currently,
>> unless
>> > you explicitly do something like set_tests_properties(.... PROPERTIES
>> > FAIL_REGULAR_EXPRESSION "Error") the tests will pass.  This seems a
>> little
>> > dangerous to me.
>> >
>> > Leo
>> >
>> > On Tue, Sep 4, 2012 at 2:58 PM, Bill Lorensen <bill.lorensen at gmail.com>
>> > wrote:
>> >>
>> >> Folks,
>> >>
>> >> VTK's test code coverage is not very good. Currently it stands at
>> >> about 62% of the code. I think the pre/post modularization coverage
>> >> rates are similar.
>> >>
>> >> One of the lessons we learned from ITK's modularization is that many
>> >> of the ITK tests were integration/regression tests versus unit
>> >> tests. Integration/regression tests tend to test multiple classes, but
>> >> fail to fully cover the code in individual classes.
>> >>
>> >> Integration/regression tests are certainly important because they test
>> >> the code as it may be used in an application.  In fact, many of VTK's
>> >> tests are derived from examples in the original text. Their purpose
>> was to
>> >> make sure the code actually did what the text said it would do.
>> >>
>> >> But, having 40% of VTK's code untested is alarming.
>> >>
>> >> Last year, I picked valgrind defects as my Fall Project:
>> >> http://itk.org/Wiki/VTK/SoftwareQuality/Valgrind/Fall2011
>> >>
>> >> I have adapted some ITK scripts to easily evaluate code coverage for
>> VTK.
>> >> There is a script in
>> >> Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh that
>> >> facilitates generating code coverage for a set of unit tests.
>> >>
>> >>
>> >> For example, running this command from your build directory
>> >> (instrumented for coverage)
>> >>
>> >> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
>> >> Domains/Chemistry -R vtkDomainsChemistry
>> >> shows that the Domains/Chemistry module has 67.6% code coverage
>> >> while
>> >> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
>> >> Geovis/Core -R vtkGeovisCore
>> >> shows that the Geovis/Core modules has 11.2% code coverage/
>> >>
>> >> This year's Fall Project is code coverage. This is a bit ironic,
>> >> because for my GE Six Sigma Project in 1995, I chose code
>> >> coverage.
>> >>
>> >> What can you do to help?
>> >> If you are adding or modifying new classes to vtk, please try to
>> >> add/enhance the tests for your class.
>> >> I'll be adding gerrit topics for tests this fall. Please help me push
>> them
>> >> through the review process
>> >>
>> >>
>> >>
>> >> Also, I'll be rejecting gerrit topics that add new classes without good
>> >> unit tests.
>> >>
>> >> Bill
>> >> '
>> >>
>> >>
>> >>
>> >> --
>> >> Unpaid intern in BillsBasement at noware dot com
>> >>
>> >>
>> >> _______________________________________________
>> >> Powered by www.kitware.com
>> >>
>> >> Visit other Kitware open-source projects at
>> >> http://www.kitware.com/opensource/opensource.html
>> >>
>> >> Follow this link to subscribe/unsubscribe:
>> >> http://www.vtk.org/mailman/listinfo/vtk-developers
>> >>
>> >>
>> >
>> >
>> > _______________________________________________
>> > Powered by www.kitware.com
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://www.vtk.org/mailman/listinfo/vtk-developers
>> >
>> >
>>
>
>
>
> --
> Unpaid intern in BillsBasement at noware dot com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20120906/4086fb99/attachment.html>