ITK Release 4/Performance Experiments/Reducing CTest Output

Testing is critical to high quality software and the ITK developers are expected to produce unit tests for each class. However, some tests produce large amounts of output which may be useful to the developer, but places a burden on the cdash database. Furthermore, some hypothesize that test output size may affect the performance of cdash.

This experiment looks at the size of output produced by ITKv4 and looks for ways to reduce a test's output.

Approach

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" test output in ITKv4. The basic methodology (from Wikipedia) consists of the following five steps:

• Define process goals that are consistent with customer demands and ITKv4's strategy.
• Measure key aspects of the current process and collect relevant data.
• Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
• Improve or optimize the process.
• Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Reduce the total test output of ITKv4 without affecting code coverage or value of the tests.

Measure

As of October 1, 2011, there were 2202 ITKv4 tests producing 10.6 meg of test output for a single platform. 2 tests produced 18% of the output and 65 of the 2202 tests produced 60% of the output. This data was gathered from a cdash file provided by Dave Cole of Kitware.

Analyze

The top ten test output producers are:

1. itkSampleToHistogramFilterTest4 the test reports failures regarding expected frequencies. A test failure is not being reported. It turns out that the test is flawed.
2. itkSystemInformationTest echoes the output of several CMake files produced during the build process, e.g. CMakeCache.txt
3. vnl_test_alignment is a third party test producing ver 400,000 characters of output.
4. itkNumericTraitsTest provides information about numeric limits and capabilities for a given platform.
5. itkImageRegionExclusionIteratorWithIndexTest provides useful information to the developer, but not necessarily for the test.
6. itkSampleToHistogramFilterTest5 provides useful information to the developer, but not necessarily for the test.
7. itkImageRegistrationMethodTest_13 produces intermediate results that are useless to the test.
8. itkSliceIteratorTest provides useful information to the developer, but not necessarily for the test.
9. itkCheckerBoardImageFilterTest produces useless output.
10. itkTriangleMeshToBinaryImageFilterTest2 produces too much output.

Analysis of these ten tests reveals the following categories of tests:

• Tests that produce valuable output, even if they pass.
• Tests that produce reasonable output < 1k characters
  • Over 1100 of the 2202 tests produce < 1k characters
• Tests that produce reasonable, but not necessarily valuable output > 1k characters
• Tests that produce erroneous output
• Tests that produce useless output
• Tests provided by Third Party software