VTK/SoftwareQuality/GerritTestFailures

Gerrit provides a powerful code review tool to the VTK and ITK open source communities. For each submitted topic, gerrit builds and tests the topic on three typical conigurations: Ubuntu, Mac and Windows. Th goal of these builds is to catch compile errors and test failures introduced by the topic, before the topic is merged into VTT of ITK.

Recurring test failures on the Gerrit builds can easily mask new defects introduced by the topic. Recently, the number of recurring failures was:

• Ubuntu - 28
• Mac - 4
• Windows - 3

This experiment seeks to reduce VTK Gerrit recurring test failures to 0 and keep them at 0.

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" to resolve these issues.

The basic methodology (from Wikipedia) consists of the following five steps:

• Define process goals that are consistent with customer demands and VTK's strategy.
• Measure key aspects of the current process and collect relevant data.
• Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
• Improve or optimize the process.
• Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Keep the number of VTK Gerrit recurring test failures(defects) to 0. When the defects stay above 0, developers find it difficult to see if their changes have increased the number of defects.

Measure

As of January 26, 2013, there were 34 defects.

• Ubuntu - 26
• Mac - 4
• Windows - 4

Analyze

Initial analysis revealed:

• Python tests accounted for 15 of the defects. Note that only the Ubuntu system has python enabled.
• One of the tests, TestSplitViewportStereoHorizontal, was failing on all three systems. It also was failing on many other nightly builds.
• One of the python tests, was failing on every VTK build that had python enabled.
• Seven tests were timing out on the Ubuntu system.
• The remaining tests required futher analysis.

Improve

• Tests timing out on Ubuntu - 7 tests
  • Increased timeouts for all tests to 61 second and 120 seconds for two tests
• Python syntax errors - 5 tests
  • Fix the tcl-to-python conversion for a few tests
• Python tests misconfiguration- 3 tests
  • Python tests missing -D argument
• Suppress recurring Mac failures - 3 tests
  • These tests were failing on virtually ever MAC
• Failed to locate materials for shaders - 3 tests
  • Some tests require VTK_USE_CG_SHADERS On. Block them if off
• NumPy tests failing - 2 tests
  • Tests that require numpy should warn if numpy is missing
• Failed image regression test - 1 test
  • http://review.source.kitware.com/#/c/9545/ Adjust error threshold for TestDensifyPolyData]
• Bad clipping planes - 1 test
  • ResetCameraClippingPlanes after manual camera setup
• TestRenderView fails on some X platforms - 1 test
  • Convert to new testing framework
• CamBlur failing with X Error of failed request: BadMatch XGetImage - 1 test
  • Convert to new testing framework
• Volume mapper failed - 1 test
  • Check for missing required extensions

Control

As of February 3, the following defects remain:

• Ubuntu - 2 defects
  • TestTemplates
  • TestSplitViewportStereoHorizontal
• Mac - 1 defect
  • TestSplitViewportStereoHorizontal
• Windows - 3 defects
  • TestSplitViewportStereoHorizontal
  • TestTilingCxx
  • TestInteractorStyleTreeMapHover

Once reduced to 0, developer diligence is needed to keep the defects to 0. The burden is on the gerrit reviewers.