ITK/Release 4/UnitTesting

ITK v4 Unit Testing Framework

Many of the existing ITK 3.20 tests were written for the purpose of testing methods and increasing code coverage. These tests accomplish this task exceedingly well, as code coverage for ITK 3.20 testing exceeds 70% of the entire toolkit. Some of the tests evaluate outputs to validate correctness of filter results. In ITK v4, the Google Testing framework will be added to assist developers in writing sound unit tests.

Unit and regression testing concepts

Formal unit and regression testing helps developers of ITK assure correct behavior of the toolkit. Characteristics of a good unit/regression test framework are:

• Tests clearly test one discrete unit of functionality
• Tests verify output(s) of code against a regression standard
• Tests are simple to read and write by developers

The Google Test framework is a well designed and thoroughly documented unit test framework. Google Test (GTest) is a natural fit with the ctest/CDash framework. GTest creates and manages individual tests, ctest executes GTest recording the results and posts to CDash. GTest is also well integrated with CMake.

Unit testing tutorial

The new unit testing framework leverages GTest and some utility functions specific for ITK. This tutorial will demonstrate many features of the new ITK testing framework (ITKTestHarness). This example show the stages of writing an ITK test, progressing from simple concepts to more complex.

Setup

In this tutorial, we are going to write a test suite or test group for the two flavors of gaussian smoothing available in ITK: [1] and [2]. The first step in the process is to create a new file in the appropriate directory. Our TestGroup will be GaussianImageFilter, so we create the file Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx. To let CMake know about our test we edit the CMakeLists.txt file in Testing/Unit/BasicFilters to include our new file:

set(BasicFiltersTestSource 
 itkGaussianImageFilterTests.cxx )

First tests

Next, we will create an edit itkGaussianImageFilterTests.cxx:

#include "itkTestHarness.h"

TEST(GaussianImageFilter,Recursive)
{
  ASSERT_TRUE ( true );
}

TEST(GaussianImageFilter,Discrete)
{
  ASSERT_TRUE ( true );
}

The first line includes the new ITK test harness.

#include "itkTestHarness.h"

We define two tests:

TEST(GaussianImageFilter,Recursive)
TEST(GaussianImageFilter,Discrete)

The TEST macro is provided by Google Test. The first argument is the TestGroup or TestSuite name. The second argument is the name of the test to run. In our case, we have added two tests called GaussianImageFilter.Recursive and GaussianImageFilter.Discrete.

Looking at the body of each test we see:

ASSERT_TRUE ( true );

ASSERT_TRUE is another Google Test macro. It's function is to ensure the body of the macro evaluates to true. If the body is true, the test continues, if the body evaluates to false, the test stops and prints out an error message. If we compile and run ctest, we see that the tests pass:

Test project /Users/blezek/Source/ITK-macosx
    Start 3: GaussianImageFilter.Discrete
3/4 Test #3: GaussianImageFilter.Discrete .....   Passed    0.01 sec
    Start 4: GaussianImageFilter.Recursive
4/4 Test #4: GaussianImageFilter.Recursive ....   Passed    0.01 sec

Under the hood, ctest is running Google Tests:

3: Test command: /Users/blezek/Source/ITK-macosx/bin/itkBasicFiltersUnitTests --gtest_filter=GaussianImageFilter.Discrete /Users/blezek/Source/ITK/Testing/Data /Users/blezek/Source/ITK-macosx/Testing/Temporary
3: Test timeout computed to be: 1500
3: Note: Google Test filter = GaussianImageFilter.Discrete
3: [==========] Running 1 test from 1 test case.
3: [----------] Global test environment set-up.
3: [----------] 1 test from GaussianImageFilter
3: [ RUN      ] GaussianImageFilter.Discrete
3: [       OK ] GaussianImageFilter.Discrete (0 ms)
3: [----------] 1 test from GaussianImageFilter (1 ms total)
3: 
3: [----------] Global test environment tear-down
3: [==========] 1 test from 1 test case ran. (1 ms total)
3: [  PASSED  ] 1 test.
3/4 Test #3: GaussianImageFilter.Discrete .....   Passed    0.01 sec

The extra output is generated by Google test.

Test failure

Now, let's see what happens if we fail a test. Modify the body of GaussianImageFilter.Recursive to be:

TEST(GaussianImageFilter,Recursive)
{
  EXPECT_TRUE ( false ) << "Continue anyway";
  ASSERT_TRUE ( false ) << "Stop running the test";
  ASSERT_TRUE ( false ) << "Never gets here";
}

This introduces the EXPECT_TRUE macro. EXPECT_TRUE evaluates it's body, if it evaluates to false, the test continues, but a failure is recorded for the test. The macro allows a text string to be piped in using the '<<' syntax to record the purpose of the test. In the code above, the test will fail on the EXPECT_TRUE, but continue to the first ASSERT_TRUE. This macro halts the test, so the second ASSERT_TRUE is never run. ASSERT's are typically used when a failure is fatal, while EXPECT's are used when the test failed, but it is safe to run the rest of the test.

When we run ctest we see:

4: [==========] Running 1 test from 1 test case.
4: [----------] Global test environment set-up.
4: [----------] 1 test from GaussianImageFilter
4: [ RUN      ] GaussianImageFilter.Recursive
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx:25: Failure
4: Value of: false
4:   Actual: false
4: Expected: true
4: Continue anyway
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx:26: Failure
4: Value of: false
4:   Actual: false
4: Expected: true
4: Stop running the test
4: [  FAILED  ] GaussianImageFilter.Recursive (0 ms)
4: [----------] 1 test from GaussianImageFilter (0 ms total)
4: 
4: [----------] Global test environment tear-down
4: [==========] 1 test from 1 test case ran. (0 ms total)
4: [  PASSED  ] 0 tests.
4: [  FAILED  ] 1 test, listed below:
4: [  FAILED  ] GaussianImageFilter.Recursive
4: 
4:  1 FAILED TEST
4/4 Test #4: GaussianImageFilter.Recursive ....***Failed    0.01 sec

In the output above, we can see the 'Continue anyway' message and the line the failure came from (line 25) as well as the 'Stop running the test' message. This helps the developer quickly understand why the test was failing.

Image test

We now edit the test to include the proper header files, and load an image. This code snippit introduces helper functions defined in the ITKTestHarness.

typedef itk::Image<float,3> FloatImage;
TEST(GaussianImageFilter,Recursive)
{
  typedef itk::RecursiveGaussianImageFilter<FloatImage, FloatImage>  GaussianFilterType;
  GaussianFilterType::Pointer filter = GaussianFilterType::New();
  FloatImage::Pointer image = LoadImage<FloatImage> ( dataFinder.GetFile ( "Input/HeadMRVolumeWithDirection.nhdr" ) );
  filter->SetInput ( image );
  ASSERT_EQ ( "", ImageSHA1Hash<FloatImage> ( filter->GetOutput() ) ) << "Failed to match the hash";
}

The first helper is dataFinder. dataFinder is a small helper class to assist in determining file names. The GetFile() method returns a filename in the ITK_DATA_ROOT directory. In this case, we would like to find Input/HeadMRVolumeWithDirection.nrdr. The next helper is LoadImage(). LoadImage is templated over the image datatype, and simply loads an image from the filename passed as an argument. These two helper functions avoid much repetitious code when writing tests.

The third helper function is ImageSHA1Hash(). This helper function computes a [SHA1 hash] of the image, and it's origin, spacing and directions. In this way, we can be sure that a filter produces exactly the same output each time it is run. If we compile and run this code, we see it fails because the hash does not match.

4: [ RUN      ] GaussianImageFilter.Recursive
4: /Users/blezek/Source/ITK/Testing/Unit/BasicFilters/itkGaussianImageFilterTests.cxx:33: Failure
4: Value of: ImageSHA1Hash<FloatImage> ( filter->GetOutput() )
4:   Actual: "93081d5322d29724e8ff49874f6b0d925adb5c1c"
4: Expected: ""
4: Failed to match the hash

If we copy the SHA1 of the image into the test, we have now documented the exact output this filter should produce each time it is run.

ASSERT_EQ ( "93081d5322d29724e8ff49874f6b0d925adb5c1c", ImageSHA1Hash<FloatImage> ( filter->GetOutput() ) ) << "Failed to match the hash";

However, the RecursiveGaussianFilter may produce different results on other platform. This will be examined in the next section.