VTK/SoftwareQuality/ContinuousBuildTestFailures

The VTK continuous build can be a powerful tool to catch defects before the nightly builds. the original purpose of the continuous build was to notify developers, vi email, if their changes introduced new defects (compile errors, warnings, failing tests). Recently, the effectiveness of the continuous build has been reduced because there are recurring continuous build errors.

Since October, 2012, the number of recurring test failures for the continuous build has varied between 7 and 11. This effort seeks to reduce VTK Continuous recurring test failures to 0 and keep them at 0.

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" to resolve these issues.

The basic methodology (from Wikipedia) consists of the following five steps:

• Define process goals that are consistent with customer demands and VTK's strategy.
• Measure key aspects of the current process and collect relevant data.
• Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
• Improve or optimize the process.
• Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Keep the number of VTK Continuous recurring test failures(defects) to 0. When the defects are above 0, developers find it difficult to notice that their changes introduce new defects.

Measure

As of February 1, 2013, there were 11 defects on the one VTK Continuous build:

• vtkChartsCoreCxx-TestColorTransferFunction
• vtkChartsCoreCxx-TestLinePlot3D
• vtkChartsCoreCxx-TestMultipleRenderers
• vtkCommonCoreTcl-otherPrint
• vtkCommonCoreTcl-TestEmptyInput
• vtkCommonCoreTcl-TestSetGet
• vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut
• vtkIOExportCxx-TestStackedPlotGL2PS-VerifyRasterizedPNG
• vtkIOMovieCxx-TestOggTheoraWriter
• vtkIOSQLCxx-SQLiteTableReadWrite
• vtkRenderingCoreCxx-TestSplitViewportStereoHorizontal

Analyze

Initial analysis revealed:

• vtkChartsCoreCxx-TestColorTransferFunction
• vtkChartsCoreCxx-TestLinePlot3D
• vtkChartsCoreCxx-TestMultipleRenderers
• vtkCommonCoreTcl-otherPrint
  • Crashing
• vtkCommonCoreTcl-TestEmptyInput
  • Crashing
• vtkCommonCoreTcl-TestSetGet
  • Crashing
• vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut
  • Image regression
• vtkIOExportCxx-TestStackedPlotGL2PS-VerifyRasterizedPNG
  • Image regression
• vtkIOMovieCxx-TestOggTheoraWriter
  • Timeout
• vtkIOSQLCxx-SQLiteTableReadWrite
  • File regression
• vtkRenderingCoreCxx-TestSplitViewportStereoHorizontal
  • Failing virtually everywhere

Control

Once reduced to 0, developer diligence is needed to keep the defects to 0. The burden is on the Gerrit reviewers.