Proposals:Increasing ITK Code Coverage
Motivation
ITK currently (Dec 20th 2008) has a 80.5% code coverage.
http://www.cdash.org/CDash/viewCoverage.php?buildid=240378
Sloccount report on number of lines of code in the Insight/Code directory returns:
158,928 lines of code
This means that about 31,000 lines of code are not tested.
We could significantly increase the code coverage of the toolkit, and in the process reduce the number of hidden bugs, by asking volunteers to adopt particular classes and write additional tests for increasing their code coverage. This could be done at the image of the "Adopt a Bug" program.
Infrastructure
It has been pointed out that the current testing infrastructure of ITK impose a high threshold of effort on contributors of new tests.
There are some existing unit test harnesses that might decrease the effort and provide additional functionality. A unit testing package for ITK must meet the following requirements:
- It must have an itk-compatible license.
- We must be able to distribute it with itk.
- It must support all itk platforms.
- It must fit within the itk test harness facility. Recall that we try to minimize the number of executables by combining large numbers of tests into FooTests.cxx files.
- It must be compatible with cmake/ctest/cdash. For example, a test must be able to "return EXIT_SUCCESS" and "return EXIT_FAILURE".
- It must not add complexity to an already complex testing process.
- It must be compatible with itk's strict attention to compile warnings and dynamic memory anlysis. In other words, it must not produce warnings or purify defects.
- It should have a minimal source footprint.
Suggestions for improving the testing system to make easier for contributors to introduce new tests include
Boost Test
Suggested by Steve Robbins
How it could work
--------------------- itkImageRegionTest.cxx ---------------------------------
#define BOOST_AUTO_TEST_MAIN
#include <boost/test/auto_unit_test.hpp>
#include "itkImageRegion.h"
template< unsigned int VImageDimension >
struct Fixture
{
typedef itk::ImageRegion<VImageDimension> RegionType;
typedef typename RegionType::IndexType IndexType;
typedef typename RegionType::SizeType SizeType;
RegionType mRegion;
};
struct Fixture1 : public Fixture<1>
{
Fixture1( int start0,
int size0 )
{
IndexType start = {{ start0 }};
SizeType size = {{ size0 }};
mRegion = RegionType( start, size );
}
};
struct Fixture2 : public Fixture<2>
{
Fixture2( int start0, int start1,
int size0, int size1 )
{
IndexType start = {{ start0, start1 }};
SizeType size = {{ size0, size1 }};
mRegion = RegionType( start, size );
}
};
struct Fixture3 : public Fixture<3>
{
Fixture3( int start0, int start1, int start2,
int size0, int size1, int size2 )
{
IndexType start = {{ start0, start1, start2 }};
SizeType size = {{ size0, size1, size2 }};
mRegion = RegionType( start, size );
}
};
BOOST_AUTO_TEST_CASE( testSlice )
{
Fixture3 volume( 12, 12, 12, 10, 20, 30 );
Fixture2 slice0( 12, 12, 20, 30 );
Fixture2 slice1( 12, 12, 10, 30 );
Fixture2 slice2( 12, 12, 10, 20 );
BOOST_CHECK_EQUAL( slice0.mRegion, volume.mRegion.Slice( 0 ) );
BOOST_CHECK_EQUAL( slice1.mRegion, volume.mRegion.Slice( 1 ) );
BOOST_CHECK_EQUAL( slice2.mRegion, volume.mRegion.Slice( 2 ) );
}
BOOST_AUTO_TEST_CASE( testSliceOutOfBounds )
{
Fixture3 volume( 12, 12, 12, 10, 20, 30 );
BOOST_CHECK_THROW( volume.mRegion.Slice( -1 ), std::exception );
BOOST_CHECK_THROW( volume.mRegion.Slice( 3 ), std::exception );
}
BOOST_AUTO_TEST_CASE( testVolumeIsInside )
{
Fixture3 volumeA( 12, 12, 12, 10, 20, 30 );
Fixture3 volumeB( 14, 14, 14, 5, 10, 15 );
BOOST_CHECK( volumeA.mRegion.IsInside( volumeB.mRegion ) );
BOOST_CHECK( ! volumeB.mRegion.IsInside( volumeA.mRegion ) );
}
--------------------- itkImageRegionTest.cxx ---------------------------------
Google Test
The Google Test framework is very similar to the Boost test harness. GTest is essentially a set of macros that assist the developer in writing concise tests. The framework does not make use of exceptions, nor templates and is supported on all major platforms and some minor ones, i.e., Cygwin, Windows CE, and Symbian. The code is available under the BSD license. The framework supports many of the features already in place through ctest, e.g., run every N'th test, run all matching tests, etc. The source code and includes are ~600K and trivially compiles using CMake without configuration or modification.
# Build Google Testing
set ( GTestSource
Utilities/gtest-1.2.1
Utilities/gtest-1.2.1/src/gtest.cc
Utilities/gtest-1.2.1/src/gtest-death-test.cc
Utilities/gtest-1.2.1/src/gtest-filepath.cc
Utilities/gtest-1.2.1/src/gtest-port.cc
Utilities/gtest-1.2.1/src/gtest-test-part.cc
Utilities/gtest-1.2.1/src/gtest-typed-test.cc
)
include_directories ( ${MI3CLib_SOURCE_DIR}/Testing/Utilities/gtest-1.2.1/include )
add_library(gtest ${BUILD_SHARED_LIBS} ${GTestSource})
Test Driver
The test driver is a very simple main function:
#include <gtest/gtest.h>
int main(int argc, char* argv[])
{
testing::InitGoogleTest ( &argc, argv );
return RUN_ALL_TESTS();
}
Types of Test
There are two basic types of tests, simple tests using the TEST macro, and test fixtures using the TEST_F macro. Many different macros are available to when executing tests, ranging from string comparisons, expected exceptions, floating point comparisons, etc. The basic framework is well documented with advanced guidance for those who dig deeper. Below is example code from an internal project that demonstrates how to write a test. Text fixtures run as methods in a sub-class of the fixture and have access to all public and protected ivars of the fixture. All test macros function as stream operators with any text directed into them appearing in the output. NB: in this example an MD5 hash is used to verify correct output rather than comparison to a known good image.
TEST(IO, LoadCT) {
mi3c::ImageLoader loader;
mi3c::Image::Pointer image = loader.setFilename ( dataFinder.getFile ( "CT.hdr" ) ).execute();
ASSERT_EQ ( "c1d43aaa5b991431a9daa1dc4b55dbb1", image->getMD5() ) << " failed to load the expected image data";
}
class ImageDataTest : public testing::Test {
public:
ImageDataTest () {
image = NULL;
floatImage = NULL;
}
virtual void SetUp() {
mi3c::ImageLoader loader;
try {
image = loader.setFilename ( dataFinder.getFile ( "MRA.hdr" ) ).execute();
mi3c::ConvertDataType convert ( mi3c::mi3cFLOAT );
floatImage = convert.execute ( image );
} catch ( itk::ImageFileReaderException e ) {
FAIL(); // Couldn't load, so fail this test before we go any further with bad data.
}
}
virtual void TearDown() {
image = NULL;
floatImage = NULL;
}
mi3c::Image::Pointer image;
mi3c::Image::Pointer floatImage;
};
TEST_F(ImageDataTest, DiscreteGaussianFilter) {
mi3c::DiscreteGaussianFilter filter;
mi3c::Image::Pointer o = filter.execute ( image );
EXPECT_EQ ( "6adeb490bda64b47e9c1bd6c547e570e", o->getMD5() ) << " Filtered with a gaussian";
EXPECT_EQ ( "300c7ee796d1b3c2b49a7649789bbf55", filter.execute ( floatImage )->getMD5() ) << " Filtered with a gaussian";
}
TEST_F(ImageDataTest, MeanFilter) {
mi3c::MeanFilter filter;
filter.setRadius ( 1 );
mi3c::Image::Pointer o = filter.execute ( image );
EXPECT_EQ ( "8b7235e1f8497b0a7fb84eb5c94af00b", o->getMD5() ) << " Mean filtered";
EXPECT_EQ ( "069a6670309db5c03a79af11a9c6e526", filter.execute ( floatImage )->getMD5() ) << " Mean filtered";
}
Running the tests
Test status is reported (in color) when running the tests and final status is reported as the exit status, much like current ITK testing.
[blezek@mi3bld04 MI3CLib-linux86-gcc]$ bin/NoOp /mi3c/projects/Source/MI3CTestData [==========] Running 9 tests from 3 test cases. [----------] Global test environment set-up. [----------] 4 tests from ImageDataTest [ RUN ] ImageDataTest.MD5 [ OK ] ImageDataTest.MD5 [ RUN ] ImageDataTest.Threshold [ OK ] ImageDataTest.Threshold [ RUN ] ImageDataTest.DiscreteGaussianFilter [ OK ] ImageDataTest.DiscreteGaussianFilter [ RUN ] ImageDataTest.MeanFilter [ OK ] ImageDataTest.MeanFilter [----------] 2 tests from IO [ RUN ] IO.LoadCT [ OK ] IO.LoadCT [ RUN ] IO.LoadInvalidFile [ OK ] IO.LoadInvalidFile [----------] 3 tests from Image [ RUN ] Image.InstantiateImage [ OK ] Image.InstantiateImage [ RUN ] Image.InstantiateImage2 [ OK ] Image.InstantiateImage2 [ RUN ] Image.TestHash [ OK ] Image.TestHash [----------] Global test environment tear-down [==========] 9 tests from 3 test cases ran. [ PASSED ] 9 tests. [blezek@mi3bld04 MI3CLib-linux86-gcc]$ echo $? 0
CMake Integration
With a slightly clever CMake macro, and a regular expression or two, Google tests are trivially integrated into CMake projects. Here, all TEST and TEST_F macros found in the source code are added as tests to the project. Each test is run as:
NoOp --gtest_filter=TestGroup.TestName
where TestGroup is the first argument to the TEST macro, and TestName is the second.
# C++ tests
set ( mi3cTestSource
Source/NoOp.cxx
Source/ImageTests.cxx
Source/IOTests.cxx
Source/FilterTests.cxx
)
add_executable(NoOp ${mi3cTestSource})
macro(ADD_GOOGLE_TESTS executable)
foreach ( source ${ARGN} )
file(READ "${source}" contents)
string(REGEX MATCHALL "TEST_?F?\\(([A-Za-z_0-9 ,]+)\\)" found_tests ${contents})
foreach(hit ${found_tests})
string(REGEX REPLACE ".*\\(([A-Za-z_0-9]+)[, ]*([A-Za-z_0-9]+)\\).*" "\\1.\\2" test_name ${hit})
add_test(${test_name} ${executable} --gtest_filter=${test_name} ${MI3CTestingDir})
endforeach(hit)
endforeach()
endmacro()
# Add all tests found in the source code, calling the executable to run them
add_google_tests ( ${EXECUTABLE_OUTPUT_PATH}/NoOp ${mi3cTestSource})
UnitTestCpp
Suggested by Mathieu Malaterre
This package is distributed under an MIT License:
