[Insight-developers] Test framework

[Luis Ibanez]

Hi Steve,

This is a very interesting suggestion.

You are right in that there is a significant burden in the process
of adding a test to ITK.

We have a lot of respect for Boost, but unfortunatly their pool
of supported compilers is smaller than what we have for ITK.

That being said, we could still build some similar infrastructure based
on CMake macros & C++ macros that (as far as I can tell) could provide
a similar functionality to what you describe.

Converting existing tests would be a significant challenge, but we could
certainly adopt a new strategy for new tests to be created from now on.

We still have 20% of the toolkit missing code coverage, so that is a
significant piece of work...  :-)

To make sure that we don't lose track of this discussion, a new Proposal
page has been created on the Wiki, under:

    http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage

The page is linked from

    http://www.itk.org/Wiki/ITK_Oversight_Committee#2008


Please feel free to add to it (or fix anything that I may have
misrepresented...).


    Regards,


       Luis


------------------------------
Steve M. Robbins wrote:
> Hello,
> 
> On Wed, Dec 10, 2008 at 03:51:56PM -0500, Bill Lorensen wrote:
> 
> 
>>Given the success of "Adopt a Bug", maybe we should initiate a "Cover a Class".
> 
> 
> Now that I have some holiday time, I'd like to raise the issue of unit
> testing frameworks.  
> 
> The few test codes that I've looked at are all written out long-hand,
> without benefit of a modern unit testing framework such as CppUnit or
> Boost.Test.  This makes writing a test case far more tedious than it
> should be and, consequently, I have never bothered (I may have sent
> /one/ to VTK).  If you truly want others to "Cover a Class", I believe
> you need to lower the barrier to writing a test case.
> 
> Would the ITK developers consider the use of Boost.Test [1]?
> 
> I'm well aware that you are justifiably reticent to add extra
> dependencies to impede the use of ITK.  I don't recall any list
> traffic about using Boost in ITK, but Boost is pretty well established
> now, portable, and is shipped with the major linux distributions.  I
> would argue that you could consider Boost as an external
> pre-requisite, akin to the requirement of a C++ compiler.  Moreover,
> if only Boost.Test is used, suitable CMake magic can be used to ensure
> the end-user doesn't need it but only those developers writing or
> running Boost.Test-based tests.
> 
> Just to be clear: I'm not suggesting that the existing tests be
> rewritten.  Nor am I advocating that Boost.Test be used for *all* new
> tests.  I'm only suggesting that Boost.Test be allowed for those of us
> who want to contribute new test cases.
> 
> So what is the advantage of Boost.Test for unit tests?  Clarity --
> which is key for test code for two reasons.  First, you want the test
> statements to stand out so you can see at a glance the logic of the
> tests.  Second, non-test boilerplate code brings with it an increased
> likelihood of bugs and you really don't want to be debugging the test
> code!
> 
> As a small example of this, consider the following snippet from
> Testing/Code/Common/itkImageRegionTest.cxx:
> 
>   try
>     {
>     SliceRegionType sliceA;
>     sliceA = regionA.Slice(2);
>     std::cout << "regionA.Slice(2): " << sliceA;
>     }
>   catch (itk::ExceptionObject &err)
>     {
>     std::cout << "Caught unexpected exception" << err;
>     return EXIT_FAILURE;
>     }
> 
>   try
>     {
>     SliceRegionType sliceA;
>     sliceA = regionA.Slice(20);
>     std::cout << "regionA.Slice(20): " << sliceA;
>     std::cout << "Failed to catch expected exception" << std::endl;
>     return EXIT_FAILURE;
>     }
>   catch (itk::ExceptionObject &err)
>     {
>     std::cout << "Caught expected exception" << err;
>     }
> 
> All of that can be replaced by:
> 
>     BOOST_CHECK_EQUAL( slice2.mRegion, volume.mRegion.Slice( 2 ) );
>     BOOST_CHECK_THROW( volume.mRegion.Slice( 20 ), std::exception );
> 
> In fact, this is not a direct replacement, but a distinct improvement:
> the first line checks the return value of the Slice() call!
> 
> At the risk of belabouring this, I'll include a Boost.Test replacement
> for the entire itkImageRegionTest.cxx.
> 
> Comments?
> 
> Best Regards,
> -Steve
> 
> [1] http://www.boost.org/doc/libs/1_37_0/libs/test/doc/html/index.html
> 
> 
> 
> --------------------- itkImageRegionTest.cxx ---------------------------------
> 
> #define BOOST_AUTO_TEST_MAIN
> #include <boost/test/auto_unit_test.hpp>
> 
> #include "itkImageRegion.h"
> 
> 
> template< unsigned int VImageDimension >
> struct Fixture
> {
>     typedef itk::ImageRegion<VImageDimension>            RegionType;
>     typedef typename RegionType::IndexType               IndexType;
>     typedef typename RegionType::SizeType                SizeType;
> 
>     RegionType mRegion;
> };
> 
> struct Fixture1 : public Fixture<1>
> {
>     Fixture1( int start0,
> 	      int size0 )
>     {
> 	IndexType start = {{ start0 }};
> 	SizeType  size  = {{ size0 }};
> 	mRegion = RegionType( start, size );
>     }
> };
> 
> struct Fixture2 : public Fixture<2>
> {
>     Fixture2( int start0, int start1,
> 	      int size0,  int size1 )
>     {
> 	IndexType start = {{ start0, start1 }};
> 	SizeType  size  = {{ size0,  size1 }};
> 	mRegion = RegionType( start, size );
>     }
> };
> 
> struct Fixture3 : public Fixture<3>
> {
>     Fixture3( int start0, int start1, int start2,
> 	      int size0,  int size1,  int size2 )
>     {
> 	IndexType start = {{ start0, start1, start2 }};
> 	SizeType  size  = {{ size0,  size1,  size2 }};
> 	mRegion = RegionType( start, size );
>     }
> };
> 
> 
> BOOST_AUTO_TEST_CASE( testSlice )
> {
>     Fixture3 volume( 12, 12, 12, 10, 20, 30 );
>     Fixture2 slice0( 12, 12, 20, 30 );
>     Fixture2 slice1( 12, 12, 10, 30 );
>     Fixture2 slice2( 12, 12, 10, 20 );
> 
>     BOOST_CHECK_EQUAL( slice0.mRegion, volume.mRegion.Slice( 0 ) );
>     BOOST_CHECK_EQUAL( slice1.mRegion, volume.mRegion.Slice( 1 ) );
>     BOOST_CHECK_EQUAL( slice2.mRegion, volume.mRegion.Slice( 2 ) );
> }
>     
> BOOST_AUTO_TEST_CASE( testSliceOutOfBounds )
> {
>     Fixture3 volume( 12, 12, 12, 10, 20, 30 );
> 
>     BOOST_CHECK_THROW( volume.mRegion.Slice( -1 ), std::exception );
>     BOOST_CHECK_THROW( volume.mRegion.Slice( 3 ), std::exception );
> }
> 
> BOOST_AUTO_TEST_CASE( testVolumeIsInside )
> {
>     Fixture3 volumeA( 12, 12, 12, 10, 20, 30 );
>     Fixture3 volumeB( 14, 14, 14,  5, 10, 15 );
> 
>     BOOST_CHECK(   volumeA.mRegion.IsInside( volumeB.mRegion ) );
>     BOOST_CHECK( ! volumeB.mRegion.IsInside( volumeA.mRegion ) );
> }
> 
> --------------------- itkImageRegionTest.cxx ---------------------------------
>