ITK/Release 4/Modularization/ Add tests: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
(Added section "Writing tests")
 
(16 intermediate revisions by 5 users not shown)
Line 1: Line 1:
The safe assumption behind testing is that "if it is not tested, it is broken".
== Non-regression tests ==
== Non-regression tests ==
The same way it used to be, except the "add_test" command:


*add_executable( [executable name] [source file or files])
The same way it used to be, except the newer syntax in "add_test" command:
 
{{#tag:syntaxhighlight
|
 
add_executable(itkBinaryThresholdFilterTest itkBinaryThresholdFilterTest.cxx)
 
target_link_libraries(itkBinaryThresholdFilterTest ${ITK-BinaryThreshold_LIBRARIES})
     
# adopt the modern CMake syntax :[NAME, COMMAND]
 
itk_add_test(NAME itkBinaryThresholdFilterTest 
        COMMAND itkBinaryThresholdFilterTest input.png output.png)
|lang=cmake}}
 
 
 
== Regression tests ==
 
=== Approach A ===
 
Using the test driver macro
The CMake macro "CreateTestDriver" defined in


*target_link_libraries([executable name] <name of libraries])
  Module/Core/TestKernal/CreateTestDriver.cmake


*adopt the modern CMake syntax :[NAME, COMMAND]: add_test(NAME [test name]  COMMAND [executable name] [arguments])
is designed for grouped regression tests in ITK.


  e.g. add_executable(itkBinaryThresholdFilterTest itkBinaryThresholdFilterTest.cxx)
All the tests in one module share one test driver.
      target_link_libraries(itkBindaryThresholdFilterTest ${ITK-BinaryThreshold_LIBRARIES})
      add_test(NAME itkBinaryThresholdFilterTest 
            COMMAND itkBinaryThresholdFilterTest input.png output.png)


{{#tag:syntaxhighlight
|


== Regression tests (need test driver): ==
  set(ITKFooFilterTests
* Approach a): using the test driver macro
    itkFooFilterTest1.cxx
The CMake macro "CreateTestDriver" defined in "Module/Core/TestKernal/CreateTestDriver.cmake" is designed to make it easy to do regression tests in ITK. All the tests in one module share one test driver.
    itkFooFilterTest2.cxx)
e.g. set(ITK-FooFilterTests
        itkFooFilterTest1.cxx
        itkFooFilterTest2.cxx)
      CreateTestDriver(ITK-FooFilter  "${ITK_FooFilter-Test_LIBRARIES}"  "${ITK-FooFilterTest}")
      add_test(NAME  itkFooFilterTest1
            COMMAND  itk-FooFilterTestDriver  itkFooFilterTest1)
      add_test(NAME  itkFooFilterTest2
            COMMAND  itk-FooFilterTestDriver  itkFooFilterTest2)


You could also find the usage of the test driver macro in any of the ITK modules (e.g. "ITK/Modules/IO/BMP/test/CMakeLists.txt")
  CreateTestDriver(ITKFooFilter  "${ITK_FooFilter-Test_LIBRARIES}"  "${ITK-FooFilterTest}")


  itk_add_test(NAME  itkFooFilterTest1
          COMMAND  ITKFooFilterTestDriver  itkFooFilterTest1)
  itk_add_test(NAME  itkFooFilterTest2
          COMMAND  ITKFooFilterTestDriver  itkFooFilterTest2)


* Approach b): stand-alone regression test
There is also a test driver executable("itkTestdriver") designed for stand-alone regression tests. For instance, all the tests in Examples are stand-alone programs each has its own main function call, in which case, the approach a) is not suitable.


e.g.    add_exectuable(itkFooTest itkFooTest.cxx)
|lang=cmake}}
        target_line_libraries( itkFooTest ${ITK_LIBRARIES})
 
        add_test(NAME  itkFooTest
 
              COMMAND  itkTestDriver
You can find Approach A used in most of the ITK modules.
for example in
 
    ITK/Modules/IO/BMP/test/CMakeLists.txt
 
=== Approach B ===
 
Stand-alone regression tests
 
There is also a test driver executable("itkTestdriver") designed for stand-alone regression tests. For instance, all the tests in Examples are stand-alone programs each has its own main function call, in which case, the approach A is not appropriate.
 
 
{{#tag:syntaxhighlight
|
   add_exectuable(itkFooTest itkFooTest.cxx)
 
  target_line_libraries( itkFooTest ${ITK_LIBRARIES})
 
  itk_add_test(NAME  itkFooTest
            COMMAND  itkTestDriver
             --compare  outputBaseline.png output.png
             --compare  outputBaseline.png output.png
                       '''$<TARGET_FILE:'''itkFooFilterTest'''>''' input.png output.png)
                       '''$<TARGET_FILE:'''itkFooFilterTest'''>''' input.png output.png)
|lang=cmake}}
The new syntax
    $<TARGET_FILE: xxx>
in the add_test command triggers CMake to locate the executable automatically. No need to specify the path for the executable here.
You can find Approach B used in Examples.
For example in
  ITK/Examples/Filtering/test/CMakeLists.txt
== Add testing data to ITK ==
  Please refer to: [http://www.itk.org/Wiki/index.php?title=ITK/Git/Develop/Data&oldid=41775 | Add Data]
== Writing tests ==
=== Unit tests ===
Writing a unit test helps ensuring that the class being exercised
* works as expected,
* works invariably along time,
* works invariably across different platforms.
It also helps in detecting potential issues, and checking how a given class is affected by changes introduced in the code.
A direct consequence of writing and executing a test is that the [http://www.vtk.org/Wiki/Special:WhatLinksHere/ITK/Code_Coverage_Practices code coverage] is improved.
When writing a unit test you should consider the following principles:
* Make sure that all methods of the class being tested are exercised.
* Make sure that if the class is templated, a number of different instances with different arguments are created so that the class is thoroughly tested.
* Make sure your test returns an EXIT_SUCCESS value when succeeding and an EXIT_FAILURE value when failing.
** When a test checks multiple features that individually may return either an EXIT_SUCCESS or an EXIT_FAILURE code, make sure that the whole test is run, and return the code in the end. This will help in locating more issues in the same run.
<source lang="cpp">
// appropriate include files
int main( int, char *[] )
{
  int testPassStatus = EXIT_SUCCESS;
  //
  // ...
  //


You could find Apprach b) used in Examples (e.g. "ITK/Examples/Filtering/CMakeLists.txt")
  if( itk::Math::FloatDifferenceULP( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) != -1 )
    {
    std::cout << "Unexpected float distance." << std::endl;
    testPassStatus = EXIT_FAILURE;
    }
  if( itk::Math::FloatAlmostEqual( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) )
    {
    std::cout << "floatRepresentationfx1 is almost equal to floatRepresentationfx2\n" << std::endl;
    }
  else
    {
    std::cout << "floatRepresentationfx1 is NOT almost equal to floatRepresentationfx2\n" << std::endl;
    testPassStatus = EXIT_FAILURE;
    }


  return testPassStatus;
}
</source>


instead of


<source lang="cpp">
// appropriate include files


int main( int, char *[] )
{
  int testPassStatus = EXIT_SUCCESS;


  //
  // ...
  //


  if( itk::Math::FloatDifferenceULP( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) != -1 )
    {
    std::cout << "Unexpected float distance." << std::endl;
    return EXIT_FAILURE;
    }
  if( itk::Math::FloatAlmostEqual( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) )
    {
    std::cout << "floatRepresentationfx1 is almost equal to floatRepresentationfx2\n" << std::endl;
    }
  else
    {
    std::cout << "floatRepresentationfx1 is NOT almost equal to floatRepresentationfx2\n" << std::endl;
    return EXIT_FAILURE;
    }


  return testPassStatus;
}
</source>


-
* When an exception can be thrown by a class, write a condition (in a try/catch block) in the test that is able to reproduce the exception condition.
* When appropriate, exercise a given method a number of times (with different arguments) so that the method is thoroughly tested.
* When required, print messages to the standard output (or error output) so that any errors happened can be checked by a human reader.
* Whenever possible, test against a baseline (ground truth): if an object is not tested against a baseline, we will not know whether it is working as expected, even if its code coverage is 100%.
* Use the testing macros defined in itkTestingMacros.h. Particularly, make sure that the macro EXERCISE_BASIC_OBJECT_METHODS is called right after the creation of the object whose class is being tested.

Latest revision as of 15:01, 16 April 2016

The safe assumption behind testing is that "if it is not tested, it is broken".

Non-regression tests

The same way it used to be, except the newer syntax in "add_test" command:

<syntaxhighlight lang="cmake">

add_executable(itkBinaryThresholdFilterTest itkBinaryThresholdFilterTest.cxx)
target_link_libraries(itkBinaryThresholdFilterTest ${ITK-BinaryThreshold_LIBRARIES})
     
# adopt the modern CMake syntax :[NAME, COMMAND]
itk_add_test(NAME itkBinaryThresholdFilterTest  
        COMMAND itkBinaryThresholdFilterTest input.png output.png)

</syntaxhighlight>


Regression tests

Approach A

Using the test driver macro The CMake macro "CreateTestDriver" defined in

 Module/Core/TestKernal/CreateTestDriver.cmake

is designed for grouped regression tests in ITK.

All the tests in one module share one test driver.

<syntaxhighlight lang="cmake">

 set(ITKFooFilterTests
    itkFooFilterTest1.cxx
    itkFooFilterTest2.cxx)
 CreateTestDriver(ITKFooFilter  "${ITK_FooFilter-Test_LIBRARIES}"  "${ITK-FooFilterTest}")
 itk_add_test(NAME  itkFooFilterTest1
          COMMAND  ITKFooFilterTestDriver  itkFooFilterTest1)

 itk_add_test(NAME  itkFooFilterTest2
          COMMAND  ITKFooFilterTestDriver  itkFooFilterTest2)


</syntaxhighlight>


You can find Approach A used in most of the ITK modules. for example in

   ITK/Modules/IO/BMP/test/CMakeLists.txt

Approach B

Stand-alone regression tests

There is also a test driver executable("itkTestdriver") designed for stand-alone regression tests. For instance, all the tests in Examples are stand-alone programs each has its own main function call, in which case, the approach A is not appropriate.


<syntaxhighlight lang="cmake">

  add_exectuable(itkFooTest itkFooTest.cxx)
  target_line_libraries( itkFooTest ${ITK_LIBRARIES})
  itk_add_test(NAME  itkFooTest
           COMMAND  itkTestDriver
           --compare  outputBaseline.png output.png
                      $<TARGET_FILE:itkFooFilterTest> input.png output.png)

</syntaxhighlight>


The new syntax

    $<TARGET_FILE: xxx> 

in the add_test command triggers CMake to locate the executable automatically. No need to specify the path for the executable here.

You can find Approach B used in Examples.

For example in

 ITK/Examples/Filtering/test/CMakeLists.txt


Add testing data to ITK

  Please refer to: | Add Data

Writing tests

Unit tests

Writing a unit test helps ensuring that the class being exercised

  • works as expected,
  • works invariably along time,
  • works invariably across different platforms.

It also helps in detecting potential issues, and checking how a given class is affected by changes introduced in the code.

A direct consequence of writing and executing a test is that the code coverage is improved. When writing a unit test you should consider the following principles:

  • Make sure that all methods of the class being tested are exercised.
  • Make sure that if the class is templated, a number of different instances with different arguments are created so that the class is thoroughly tested.
  • Make sure your test returns an EXIT_SUCCESS value when succeeding and an EXIT_FAILURE value when failing.
    • When a test checks multiple features that individually may return either an EXIT_SUCCESS or an EXIT_FAILURE code, make sure that the whole test is run, and return the code in the end. This will help in locating more issues in the same run.

<source lang="cpp"> // appropriate include files

int main( int, char *[] ) {

 int testPassStatus = EXIT_SUCCESS;
 //
 // ...
 //
 if( itk::Math::FloatDifferenceULP( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) != -1 )
   {
   std::cout << "Unexpected float distance." << std::endl;
   testPassStatus = EXIT_FAILURE;
   }
 if( itk::Math::FloatAlmostEqual( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) )
   {
   std::cout << "floatRepresentationfx1 is almost equal to floatRepresentationfx2\n" << std::endl;
   }
 else
   {
   std::cout << "floatRepresentationfx1 is NOT almost equal to floatRepresentationfx2\n" << std::endl;
   testPassStatus = EXIT_FAILURE;
   }
 return testPassStatus;

} </source>

instead of

<source lang="cpp"> // appropriate include files

int main( int, char *[] ) {

 int testPassStatus = EXIT_SUCCESS;
 //
 // ...
 //
 if( itk::Math::FloatDifferenceULP( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) != -1 )
   {
   std::cout << "Unexpected float distance." << std::endl;
   return EXIT_FAILURE;
   }
 if( itk::Math::FloatAlmostEqual( floatRepresentationfx1.asFloat, floatRepresentationfx2.asFloat ) )
   {
   std::cout << "floatRepresentationfx1 is almost equal to floatRepresentationfx2\n" << std::endl;
   }
 else
   {
   std::cout << "floatRepresentationfx1 is NOT almost equal to floatRepresentationfx2\n" << std::endl;
   return EXIT_FAILURE;
   }
 return testPassStatus;

} </source>

  • When an exception can be thrown by a class, write a condition (in a try/catch block) in the test that is able to reproduce the exception condition.
  • When appropriate, exercise a given method a number of times (with different arguments) so that the method is thoroughly tested.
  • When required, print messages to the standard output (or error output) so that any errors happened can be checked by a human reader.
  • Whenever possible, test against a baseline (ground truth): if an object is not tested against a baseline, we will not know whether it is working as expected, even if its code coverage is 100%.
  • Use the testing macros defined in itkTestingMacros.h. Particularly, make sure that the macro EXERCISE_BASIC_OBJECT_METHODS is called right after the creation of the object whose class is being tested.