VTK/SoftwareQuality/ContinuousBuildTestFailures

The VTK continuous build can be a powerful tool to catch defects before the nightly builds. the original purpose of the continuous build was to notify developers, vi email, if their changes introduced new defects (compile errors, warnings, failing tests). Recently, the effectiveness of the continuous build has been reduced because there are recurring continuous build errors.

Since October, 2012, the number of recurring test failures for the continuous build has varied between 7 and 11. This effort seeks to reduce VTK Continuous recurring test failures to 0 and keep them at 0.

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" to resolve these issues.

The basic methodology (from Wikipedia) consists of the following five steps:

• Define process goals that are consistent with customer demands and VTK's strategy.
• Measure key aspects of the current process and collect relevant data.
• Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
• Improve or optimize the process.
• Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Keep the number of VTK Continuous recurring test failures(defects) to 0. When the defects are above 0, developers find it difficult to notice that their changes introduce new defects.

Measure

As of February 1, 2013, there were 11 defects on the one VTK Continuous build:

• vtkChartsCoreCxx-TestColorTransferFunction
• vtkChartsCoreCxx-TestLinePlot3D
• vtkChartsCoreCxx-TestMultipleRenderers
• vtkCommonCoreTcl-otherPrint
• vtkCommonCoreTcl-TestEmptyInput
• vtkCommonCoreTcl-TestSetGet
• vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut
• vtkIOExportCxx-TestStackedPlotGL2PS-VerifyRasterizedPNG
• vtkIOMovieCxx-TestOggTheoraWriter
• vtkIOSQLCxx-SQLiteTableReadWrite
• vtkRenderingCoreCxx-TestSplitViewportStereoHorizontal

Analyze

Initial analysis revealed:

• vtkChartsCoreCxx-TestColorTransferFunction
  • Image regression
• vtkChartsCoreCxx-TestLinePlot3D
  • Image regression
• vtkChartsCoreCxx-TestMultipleRenderers
  • Invalid test results
• vtkCommonCoreTcl-otherPrint
  • Crashing
• vtkCommonCoreTcl-TestEmptyInput
  • Crashing
• vtkCommonCoreTcl-TestSetGet
  • Crashing
• vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut
  • Image regression
• vtkIOExportCxx-TestStackedPlotGL2PS-VerifyRasterizedPNG
  • Image regression
• vtkIOMovieCxx-TestOggTheoraWriter
  • Timeout
• vtkIOSQLCxx-SQLiteTableReadWrite
  • File regression
• vtkRenderingCoreCxx-TestSplitViewportStereoHorizontal
  • Failing virtually everywhere

Improve

Gerrit reviews

• vtkChartsCoreCxx-TestColorTransferFunction
  • Added additional Baseline
• vtkChartsCoreCxx-TestLinePlot3D
  • Added additional Baseline
• vtkChartsCoreCxx-TestMultipleRenderers
  • Fix TestMultipleRenderers on Dash3's continuous

Apparently this test has been failing on dash3 since October. This problem is ultimately due to the fact that we've enabled the depth buffer within vtkOpenGLContextDevice3D::Begin(). This commit changes this behavior, so we now enable the depth buffer at the beginning of each 3D context "Draw" method, and then disable it at the end.

While investigating this problem, I realized that we copied code from vtkOpenGLContextDevice2D and pasted it into the new 3D device class. It turns out that this is unnecessary, so this commit removes the copied & pasted code from vtkOpenGLContextDevice3D.

• vtkCommonCoreTcl-otherPrint
  • Crashing
• vtkCommonCoreTcl-TestEmptyInput
  • Crashing
• vtkCommonCoreTcl-TestSetGet
  • BUG: TestSetGet crashes

TestSetGet tries to test the singleton vtkTextRenderer. This causes segfaults on some systems. This patch exempts vtkTextRenderer from the test.

• vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DCut
  • Image regression
• vtkIOExportCxx-TestStackedPlotGL2PS-VerifyRasterizedPNG
  • Image regression
• vtkIOMovieCxx-TestOggTheoraWriter
  • Timeout
• vtkIOSQLCxx-SQLiteTableReadWrite
  • File regression
• vtkRenderingCoreCxx-TestSplitViewportStereoHorizontal
  • Turn off multisamples so antialiasing doesn't cause test failures

TestSplitViewportStereoHorizontal is failing on several platforms.

Some linux platforms failed because their graphics cards don't do antialiasing. This commit turns of antialiasing on ALL platforms provides a new non-antialiased baseline, and increases the error threshold.

All windows machines failed because SetStereoRender was called before the StereoType was set. This commit should fix that, because it calls SetStereoType before SetStereoRender.

All OS X machines still fail because the ViewAngle is, for some unknown reason, not interpreted properly for this test on that platform.

Control

Once reduced to 0, developer diligence is needed to keep the defects to 0. The burden is on the Gerrit reviewers.