ITK/Release 4/UnitTesting

ITK v4 Unit Testing Framework

Many of the existing ITK 3.20 tests were written for the purpose of testing methods and increasing code coverage. These tests accomplish this task exceedingly well, as code coverage for ITK 3.20 testing exceeds 70% of the entire toolkit. Some of the tests evaluate outputs to validate correctness of filter results. In ITK v4, the Google Testing framework will be added to assist developers in writing sound unit tests.

Unit and regression testing concepts

Formal unit and regression testing helps developers of ITK assure correct behavior of the toolkit. Characteristics of a good unit/regression test framework are:

• Tests clearly test one discrete unit of functionality
• Tests verify output(s) of code against a regression standard
• Tests are simple to read and write by developers

The Google Test framework is a well designed and thoroughly documented unit test framework. Google Test (GTest) is a natural fit with the ctest/CDash framework. GTest creates and manages individual tests, ctest executes GTest recording the results and posts to CDash. GTest is also well integrated with CMake.

Unit testing proposal

An integrated Google Test framework has been proposed for ITK v4. This combines Google Testing with ITK specific utilities to aid the developer in writing solid unit tests. A draft implementation has been completed and is available on Github in a topic branch (

Unit testing basics tutorial

The new unit testing framework leverages GTest and some utility functions specific for ITK. This tutorial will demonstrate many features of the new ITK testing framework (ITKTestHarness). This example show the stages of writing an ITK test, progressing from simple concepts to more complex.

Setup

In this tutorial, we are going to write a test suite or test group for recursive gaussian smoothing available in ITK (itkRecursiveGaussianImageFilter). The first step in the process is to create a new file in the appropriate directory. Our TestGroup will be GaussianImageFilter, so we create the file Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx. To let CMake know about our test we edit the CMakeLists.txt file in Testing/Unit/BasicFilters to include our new file:

set(BasicFiltersTestSource 
 itkRecursiveGaussianImageFilterUnitTests.cxx )

First tests

Next, we will create an edit itkRecursiveGaussianImageFilterUnitTests.cxx:

#include "itkTestHarness.h"

TEST(RecursiveGaussianImageFilter,Basics)
{
  ASSERT_TRUE ( true );
}

The first line includes the new ITK test harness.

#include "itkTestHarness.h"

We define a test:

TEST(RecursiveGaussianImageFilter,Basics)

The TEST macro is provided by Google Test. The first argument is the TestGroup or TestSuite name. The second argument is the name of the test to run. In our case, we have added a test called RecursiveGaussianImageFilter.Basics

Looking at the body of the test we see:

ASSERT_TRUE ( true );

ASSERT_TRUE is another Google Test macro. It's function is to ensure the body of the macro evaluates to true. If the body is true, the test continues, if the body evaluates to false, the test stops and prints out an error message. If we compile and run ctest, we see that the tests pass:

Test project /Users/blezek/Source/ITK-macosx
    Start 3: RecursiveGaussianImageFilter.Basics
3/4 Test #3: RecursiveGaussianImageFilter.Basics .......   Passed    0.01 sec

Under the hood, ctest is running Google Tests:

3: Test command: /Users/blezek/Source/ITK-macosx/bin/itkBasicFiltersUnitTests --gtest_filter=RecursiveGaussianImageFilter.Basics /Users/blezek/Source/ITK/Testing/Data /Users/blezek/Source/ITK-macosx/Testing/Temporary
3: Test timeout computed to be: 1500
3: Note: Google Test filter = RecursiveGaussianImageFilter.Basics
3: [==========] Running 1 test from 1 test case.
3: [----------] Global test environment set-up.
3: [----------] 1 test from GaussianImageFilter
3: [ RUN      ] RecursiveGaussianImageFilter.Basics
3: [       OK ] RecursiveGaussianImageFilter.Basics (0 ms)
3: [----------] 1 test from RecursiveGaussianImageFilter (1 ms total)
3: 
3: [----------] Global test environment tear-down
3: [==========] 1 test from 1 test case ran. (1 ms total)
3: [  PASSED  ] 1 test.
3/4 Test #3: RecursiveGaussianImageFilter.Basics .....   Passed    0.01 sec

The extra output is generated by Google test.

Test failure

Now, let's see what happens if we fail a test. Modify the body of RecursiveGaussianImageFilter.Basics to be:

TEST(RecursiveGaussianImageFilter,Basics)
{
  EXPECT_TRUE ( false ) << "Continue anyway";
  ASSERT_TRUE ( false ) << "Stop running the test";
  ASSERT_TRUE ( false ) << "Never gets here";
}

This introduces the EXPECT_TRUE macro. EXPECT_TRUE evaluates it's body, if it evaluates to false, the test continues, but a failure is recorded for the test. The macro allows a text string to be piped in using the '<<' syntax to record the purpose of the test. In the code above, the test will fail on the EXPECT_TRUE, but continue to the first ASSERT_TRUE. This macro halts the test, so the second ASSERT_TRUE is never run. ASSERT's are typically used when a failure is fatal, while EXPECT's are used when the test failed, but it is safe to run the rest of the test.

When we run ctest we see:

4: [==========] Running 1 test from 1 test case.
4: [----------] Global test environment set-up.
4: [----------] 1 test from RecursiveGaussianImageFilter
4: [ RUN      ] RecursiveGaussianImageFilter.Basics
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkRecursiveGaussianImageFilterUnitTests.cxx:25: Failure
4: Value of: false
4:   Actual: false
4: Expected: true
4: Continue anyway
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkRecursiveGaussianImageFilterUnitTests.cxx:26: Failure
4: Value of: false
4:   Actual: false
4: Expected: true
4: Stop running the test
4: [  FAILED  ] RecursiveGaussianImageFilter.Basics (0 ms)
4: [----------] 1 test from RecursiveGaussianImageFilter (0 ms total)
4: 
4: [----------] Global test environment tear-down
4: [==========] 1 test from 1 test case ran. (0 ms total)
4: [  PASSED  ] 0 tests.
4: [  FAILED  ] 1 test, listed below:
4: [  FAILED  ] RecursiveGaussianImageFilter.Basics
4: 
4:  1 FAILED TEST
4/4 Test #4: GRecursiveGaussianImageFilter.Basics ....***Failed    0.01 sec

In the output above, we can see the 'Continue anyway' message and the line the failure came from (line 25) as well as the 'Stop running the test' message. This helps the developer quickly understand why the test was failing.

Image test

We now edit the test to include the proper header files, and load an image. This code snippit introduces helper functions defined in the ITKTestHarness.

typedef itk::Image<float,3> FloatImage;
TEST(RecursiveGaussianImageFilter,Basics)
{
  typedef itk::RecursiveGaussianImageFilter<FloatImage, FloatImage>  GaussianFilterType;
  GaussianFilterType::Pointer filter = GaussianFilterType::New();
  FloatImage::Pointer image = LoadImage<FloatImage> ( dataFinder.GetFile ( "Input/HeadMRVolumeWithDirection.nhdr" ) );
  filter->SetInput ( image );
  ASSERT_EQ ( "", ImageSHA1Hash<FloatImage> ( filter->GetOutput() ) ) << "Failed to match the hash";
}

The first helper is dataFinder. dataFinder is a small helper class to assist in determining file names. The GetFile() method returns a filename in the ITK_DATA_ROOT directory. In this case, we would like to find Input/HeadMRVolumeWithDirection.nrdr. The next helper is LoadImage(). LoadImage is templated over the image datatype, and simply loads an image from the filename passed as an argument. These two helper functions avoid much repetitious code when writing tests.

The third helper function is ImageSHA1Hash(). This helper function computes a [SHA1 hash] of the image, and it's origin, spacing and directions. In this way, we can be sure that a filter produces exactly the same output each time it is run. If we compile and run this code, we see it fails because the hash does not match.

4: [ RUN      ] RecursiveGaussianImageFilter.Basics
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkRecursiveGaussianImageFilterUnitTests.cxx:33: Failure
4: Value of: ImageSHA1Hash<FloatImage> ( filter->GetOutput() )
4:   Actual: "93081d5322d29724e8ff49874f6b0d925adb5c1c"
4: Expected: ""
4: Failed to match the hash

If we copy the SHA1 of the image into the test, we have now documented the exact output this filter should produce each time it is run.

ASSERT_EQ ( "93081d5322d29724e8ff49874f6b0d925adb5c1c", ImageSHA1Hash<FloatImage> ( filter->GetOutput() ) ) << "Failed to match the hash";

However, the RecursiveGaussianFilter may produce different results on other platform. This will be examined in the next section.

Image comparison

Different platforms have differing levels of numeric precision. Because of this, a SHA1 hash value will not be valid on all platforms unless the filter is designed to produce exactly the same output independent of numeric precision (morphology filters for instance). The testing harness provides several helper macros that operate exactly like the Google Test macros. These helpers operate on images and require a few extra bits of information.

  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "float", 1.0);
  filter->SetSigma( 2.0 );
  filter->SetZeroOrder();
  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "sigma_2.0", 1.0);

The macros are:

#define EXPECT_IMAGE_EQ(ImageType, image, name)
#define EXPECT_IMAGE_NEAR(ImageType, image, name, tolerance)
#define ASSERT_IMAGE_EQ(ImageType, image, name)
#define ASSERT_IMAGE_NEAR(ImageType, image, name, tolerance)

The first argument is the type of the image and is used to instantiate the correct reading and comparison classes. The image argument is the actual image produced during this run of the test. The name specifies which image this is. Generally speaking, a test may execute a filter several times producing different images each time. The name is used to construct the baseline image file on disk. The tolerance argument for the "_NEAR" functions provide a threshold for the image difference from baseline. Running the test at this stage produces this output.

3: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkRecursiveGaussianImageFilterUnitTests.cxx:33: Failure
3: Value of: imageCompare.Compare (filter->GetOutput(), "float", 1.0)
3:   Actual: false
3: Expected: true
3: Baseline does not exist, wrote /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/RecursiveGaussianImageFilter_Basics_float.nrrd
3: cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/RecursiveGaussianImageFilter_Basics_float.nrrd /Users/blezek/Source/ITK/Testing/Data//Baseline/RecursiveGaussianImageFilter_Basics_float.nrrd
3: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkRecursiveGaussianImageFilterUnitTests.cxx:36: Failure
3: Value of: imageCompare.Compare (filter->GetOutput(), "sigma_2.0", 1.0)
3:   Actual: false
3: Expected: true
3: Baseline does not exist, wrote /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/RecursiveGaussianImageFilter_Basics_sigma_2.0.nrrd
3: cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/RecursiveGaussianImageFilter_Basics_sigma_2.0.nrrd /Users/blezek/Source/ITK/Testing/Data//Baseline/RecursiveGaussianImageFilter_Basics_sigma_2.0.nrrd
3: [  FAILED  ] RecursiveGaussianImageFilter.Basics (69 ms)
3: [----------] 1 test from RecursiveGaussianImageFilter (69 ms total)
3: 
3: [----------] Global test environment tear-down
3: [==========] 1 test from 1 test case ran. (69 ms total)
3: [  PASSED  ] 0 tests.
3: [  FAILED  ] 1 test, listed below:
3: [  FAILED  ] RecursiveGaussianImageFilter.Basics
3: 
3:  1 FAILED TEST
3/3 Test #3: RecursiveGaussianImageFilter.Basics ...***Failed    0.09 sec

On the first run, the test failed because no baseline existed. EXPECT_IMAGE_NEAR writes the image to disk in the temporary directory. Since we used the EXPECT version of the macro, the test continued and wrote the second baseline image. The naming convention for the files in TestGroup_Test_name.png; this example wrote GaussianImageFilter_Recursive_float.nrrd and GaussianImageFilter_Recursive_sigma_2.0.nrrd. If we take a look at the image we see (slice 16):

The image is fairly small because the input image is very small. After we examine these images, we can copy them to the Baseline image directory in ITK_DATA_ROOT and make them available for other to use. The macro helpfully supplies the proper command line:

cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/RecursiveGaussianImageFilter_Basics_float.nrrd /Users/blezek/Source/ITK/Testing/Data//Baseline/RecursiveGaussianImageFilter_Basics_float.nrrd
cp /Users/blezek/Source/ITK-macosx/Testing/Temporary/Newbaseline/RecursiveGaussianImageFilter_Basics_sigma_2.0.nrrd /Users/blezek/Source/ITK/Testing/Data//Baseline/RecursiveGaussianImageFilter_Basics_sigma_2.0.nrrd

Now when we run the testing, the test passes.

Test Fixtures

Next, let's expand on our simple basics test and create a text fixture. If a series of tests share similar setup and/or tear down code, it is simple to place them in a fixture. In Google Test, a fixture is a class derived from ::testing::Test. The new class can implement a SetUp() and TearDown() method. Let's see what this looks like.

RecursiveGaussianImageFilter test fixture

Suppose we want to test a gaussian filter on both short and float image data types. We have a number of tests that we would like to run that all follow the pattern: load images, set parameters, compare results. Test fixtures reduce the code duplication by moving initialization code into a SetUp() method that is called before each test run.

The fixture class looks like this:

typedef itk::Image< float, 3 > FloatImage;
typedef itk::Image< short, 3 > ShortImage;
typedef itk::RecursiveGaussianImageFilter< FloatImage, FloatImage > GaussianFilterType;
typedef itk::RecursiveGaussianImageFilter< ShortImage, ShortImage > GaussianFilterTypeShort;

class RecursiveGaussianImageFilter : public ::testing::Test
{
public:
  virtual void SetUp()
  {
    filter = GaussianFilterType::New();
    filterShort = GaussianFilterTypeShort::New();
    EXPECT_NO_THROW ( floatImage = LoadImage<FloatImage> ( dataFinder.GetFile ( "Input/HeadMRVolumeWithDirection.nhdr" ) ) ) << "Failed to load float volume";
    EXPECT_NO_THROW ( shortImage = LoadImage<ShortImage> ( dataFinder.GetFile ( "Input/HeadMRVolumeWithDirection.nhdr" ) ) ) << "Failed to load short volume";
  }
  GaussianFilterType::Pointer filter;
  GaussianFilterTypeShort::Pointer filterShort
  FloatImage::Pointer floatImage;
  ShortImage::Pointer shortImage;
};

The first thing to notice is that RecursiveGaussianImageFilter derives from ::testing::Test. In the SetUp() method, we have now wrapped our LoadImage() call in the Google Test macro EXPECT_NO_THROW(). As you might guess, EXPECT_NO_THROW() runs the code in the body and catches any thrown exceptions. In this way, we ensure that our tests have good data to work on, or they fail in the SetUp() call.

Now we change our test.

TEST_F(RecursiveGaussianImageFilter, FloatBasics)
{
  filter->SetInput (floatImage);
  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "float", 1.0);
  filter->SetSigma(2.0);
  filter->SetZeroOrder();
  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "sigma_2.0", 1.0);
}

Instead of the TEST() macro, we now use TEST_F(). TEST_F() tells Google Test that this test runs in the fixture specified by the test group (RecursiveGaussianImageFilter). This macro generates an error if the class RecursiveGaussianImageFilter does not exist, or does not publicly derive from ::testing::Test. The test body is the same as before, except we operate on floatImage, and no longer need to load the data inside the test body. The baseline image name is derived in exactly the same way as before. More details on test fixtures can be found in the Google Test documentation.

We can define several other tests in the same fixture:

TEST_F(RecursiveGaussianImageFilter, ShortBasics)
{
  filterShort->SetInput (shortImage);
  EXPECT_IMAGE_NEAR(ShortImage, filterShort->GetOutput(), "short", 1.0);
  filterShort->SetSigma(2.0);
  filterShort->SetZeroOrder();
  EXPECT_IMAGE_NEAR(ShortImage, filterShort->GetOutput(), "sigma_2.0", 1.0);
}

TEST_F(RecursiveGaussianImageFilter,ZSmoothing)
{
  filter->SetDirection( 2 );  // apply along Z
  filter->SetOrder( GaussianFilterType::ZeroOrder );
  filter->SetInput ( floatImage );
  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "ZeroOrder", 1.0);
  filter->SetOrder ( GaussianFilterType::FirstOrder );
  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "FirstOrder", 1.0);
  filter->SetOrder ( GaussianFilterType::SecondOrder );
  EXPECT_IMAGE_NEAR(FloatImage, filter->GetOutput(), "SecondOrder", 1.0);
}

One thing to remember: since CMake is responsible for finding tests in the framework, if you add a test to an existing file, you must re-run CMake to have the test executed during the next run of ctest. In practice this is rarely a problem.