[vtk-developers] VTK test code coverage

[Pat Marion]

On another project, we developed a test helper that allowed this type of
test:

// expect two errors
testHelper->StartExpectingErrors(2);

myClass->DoSomethingWrong();
myClass->ThisIsAlsoWrong();

testHelper->StopExpectingErrors();


At the end of the test, it would be a failure if there weren't exactly two
error macros triggered between the Start/Stop calls.  Any errors/warning
macros that are triggered elsewhere will result in test failure.  We found
this to be crucial to catching regressions.  In our experience, it was
quite common for a commit to trigger new error macros, while the test would
have otherwise continued to pass.

Pat

On Wed, Sep 5, 2012 at 3:47 PM, David Cole <david.cole at kitware.com> wrote:

> I have always thought it should be a test failure (by default) if the
> output contains the error string from encountering a vtkError while
> running the test. Any exceptions to that rule would be simply to test
> the error mechanism itself.
>
> I know there are some tests that do produce Error messages when I run
> them on my machine...
>
> Seems like those should be test failures, but some are not.
>
>
> On Wed, Sep 5, 2012 at 2:46 PM, Yuanxin Liu <leo.liu at kitware.com> wrote:
> > Hi, Bill,
> >   A separate but related question:  What should be the default behavior
> of
> > tests regarding "ERROR" strings in the standard output?  Currently,
> unless
> > you explicitly do something like set_tests_properties(.... PROPERTIES
> > FAIL_REGULAR_EXPRESSION "Error") the tests will pass.  This seems a
> little
> > dangerous to me.
> >
> > Leo
> >
> > On Tue, Sep 4, 2012 at 2:58 PM, Bill Lorensen <bill.lorensen at gmail.com>
> > wrote:
> >>
> >> Folks,
> >>
> >> VTK's test code coverage is not very good. Currently it stands at
> >> about 62% of the code. I think the pre/post modularization coverage
> >> rates are similar.
> >>
> >> One of the lessons we learned from ITK's modularization is that many
> >> of the ITK tests were integration/regression tests versus unit
> >> tests. Integration/regression tests tend to test multiple classes, but
> >> fail to fully cover the code in individual classes.
> >>
> >> Integration/regression tests are certainly important because they test
> >> the code as it may be used in an application.  In fact, many of VTK's
> >> tests are derived from examples in the original text. Their purpose was
> to
> >> make sure the code actually did what the text said it would do.
> >>
> >> But, having 40% of VTK's code untested is alarming.
> >>
> >> Last year, I picked valgrind defects as my Fall Project:
> >> http://itk.org/Wiki/VTK/SoftwareQuality/Valgrind/Fall2011
> >>
> >> I have adapted some ITK scripts to easily evaluate code coverage for
> VTK.
> >> There is a script in
> >> Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh that
> >> facilitates generating code coverage for a set of unit tests.
> >>
> >>
> >> For example, running this command from your build directory
> >> (instrumented for coverage)
> >>
> >> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
> >> Domains/Chemistry -R vtkDomainsChemistry
> >> shows that the Domains/Chemistry module has 67.6% code coverage
> >> while
> >> ../VTK/Utilities/Maintenance/computeCodeCoverageLocallyForOneTest.sh
> >> Geovis/Core -R vtkGeovisCore
> >> shows that the Geovis/Core modules has 11.2% code coverage/
> >>
> >> This year's Fall Project is code coverage. This is a bit ironic,
> >> because for my GE Six Sigma Project in 1995, I chose code
> >> coverage.
> >>
> >> What can you do to help?
> >> If you are adding or modifying new classes to vtk, please try to
> >> add/enhance the tests for your class.
> >> I'll be adding gerrit topics for tests this fall. Please help me push
> them
> >> through the review process
> >>
> >>
> >>
> >> Also, I'll be rejecting gerrit topics that add new classes without good
> >> unit tests.
> >>
> >> Bill
> >> '
> >>
> >>
> >>
> >> --
> >> Unpaid intern in BillsBasement at noware dot com
> >>
> >>
> >> _______________________________________________
> >> Powered by www.kitware.com
> >>
> >> Visit other Kitware open-source projects at
> >> http://www.kitware.com/opensource/opensource.html
> >>
> >> Follow this link to subscribe/unsubscribe:
> >> http://www.vtk.org/mailman/listinfo/vtk-developers
> >>
> >>
> >
> >
> > _______________________________________________
> > Powered by www.kitware.com
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Follow this link to subscribe/unsubscribe:
> > http://www.vtk.org/mailman/listinfo/vtk-developers
> >
> >
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://www.vtk.org/mailman/listinfo/vtk-developers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/vtk-developers/attachments/20120905/1dfea526/attachment.html>