ITK Release 4/Performance Experiments/Reducing CTest Output

Testing is critical to high quality software and the ITK developers are expected to produce unit tests for each class. However, some tests produce large amounts of output which may be useful to the developer, but places a burden on the cdash database. Furthermore, some hypothesize that test output size may affect the performance of cdash.

This experiment looks at the size of output produced by ITKv4 and looks for ways to reduce a test's output.

Approach

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" test output in ITKv4. The basic methodology (from Wikipedia) consists of the following five steps:

• Define process goals that are consistent with customer demands and ITKv4's strategy.
• Measure key aspects of the current process and collect relevant data.
• Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
• Improve or optimize the process.
• Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Reduce the total test output of ITKv4 without affecting code coverage or value of the tests.

Measure

As of October 1, 2011, there were 2202 ITKv4 tests producing 10.6 meg of test output for a single platform. 2 tests produced 18% of the output and 65 of the 2202 tests produced 60% of the output. This data was gathered from a cdash file provided by Dave Cole of Kitware.

Analyze

Analysis of test output sizes revealed the following types of tests:

• Tests that produce valuable output, even if they pass:
  • itkSystemInformation - echoes the output of several CMake files produced during the build process, e.g. CMakeCache.txt
  • itkNumericsTests - provides information about numeric limits and capabilities for a given platform.
• Tests that produce reasonable output < 1k characters
  • Over 1100 of the 2202 tests produce < 1k characters
• Tests that produce reasonable, but not necessary valuable output > 1k characters
• Tests that produce erroneous output
  • SampleToHistogramFilterTest4 - the test reports failures regarding expected frequencies. A test failure is not being reported. It turns out that the test is flawed.
• Tests that produce useless output
  • SampleToHistogramFilterTest5 produces way too much useless output.
  • itkSimplexMesh PrintSelf produces too much output.
  • CheckerBoardImageTest produces useless output
  • itkImageRegistrationMethodTest_13 produces intermediate results that are useless to the test.
• Tests provided by Third Party software
  • vnl_alignment produces >400,000 characters