VTK/SoftwareQuality/ContinuousBuildTestFailures

The VTK continuous build can be a powerful tool to catch defects before the nightly builds.

Recurring test failures on the VTK continuous build reduce the effectiveness of this facility.

Since October, 2012, the number of recurring test failures for the continuous build has varied between 7 and 11.

This experiment seeks to reduce VTK Continuous recurring test failures to 0 and keep them at 0.

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" to resolve these issues.

The basic methodology (from Wikipedia) consists of the following five steps:

• Define process goals that are consistent with customer demands and VTK's strategy.
• Measure key aspects of the current process and collect relevant data.
• Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
• Improve or optimize the process.
• Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Keep the number of VTK Continuous recurring test failures(defects) to 0. When the defects are above 0, developers find it difficult to see if their changes have increased the number of defects.

Measure

As of February 1, 2013, there were 11 defects on the one VTK Continuous build:

• vtkChartsCoreCxx-TestColorTransferFunction
• vtkChartsCoreCxx-TestLinePlot3D
• vtkChartsCoreCxx-TestMultipleRenderers
• vtkCommonCoreTcl-otherPrint
• vtkCommonCoreTcl-TestEmptyInput
• vtkCommonCoreTcl-TestSetGet
• vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut
• vtkIOExportCxx-TestStackedPlotGL2PS-VerifyRasterizedPNG
• vtkIOMovieCxx-TestOggTheoraWriter
• vtkIOSQLCxx-SQLiteTableReadWrite
• vtkRenderingCoreCxx-TestSplitViewportStereoHorizontal

Analyze

Initial analysis revealed:

• vtkChartsCoreCxx-TestColorTransferFunction
• vtkChartsCoreCxx-TestLinePlot3D
• vtkChartsCoreCxx-TestMultipleRenderers
• vtkCommonCoreTcl-otherPrint
• vtkCommonCoreTcl-TestEmptyInput
• vtkCommonCoreTcl-TestSetGet
• vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut
• vtkIOExportCxx-TestStackedPlotGL2PS-VerifyRasterizedPNG
• vtkIOMovieCxx-TestOggTheoraWriter
• vtkIOSQLCxx-SQLiteTableReadWrite
• vtkRenderingCoreCxx-TestSplitViewportStereoHorizontal

Control

Once reduced to 0, developer diligence is needed to keep the defects to 0. The burden is on the Gerrit reviewers.