VTK/SoftwareQuality/NightlyExpectedBuildTestFailures: Difference between revisions
| Line 71: | Line 71: | ||
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped] | [http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped] | ||
| 1 | | 1 | ||
| | | [http://open.cdash.org/testDetails.php?test=178384129&build=2823951 Image regression] | ||
AIX exit | |||
| Unassigned | | Unassigned | ||
|- | |- | ||
| Line 77: | Line 78: | ||
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingParallelCxx-MPI-TestClientServerRendering vtkRenderingParallelCxx-MPI-TestClientServerRendering] | [http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingParallelCxx-MPI-TestClientServerRendering vtkRenderingParallelCxx-MPI-TestClientServerRendering] | ||
| 1 | | 1 | ||
| | | [http://open.cdash.org/testDetails.php?test=178354074&build=2823558 Failed to connect] | ||
| Unassigned | | Unassigned | ||
|- | |- | ||
| Line 89: | Line 90: | ||
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy] | [http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy vtkInfovisLayoutCxx-TestCirclePackLayoutStrategy] | ||
| 1 | | 1 | ||
| | | [http://open.cdash.org/testDetails.php?test=178377163&build=2823849 Major regression failure] | ||
| Unassigned | | Unassigned | ||
|- | |- | ||
| Line 101: | Line 102: | ||
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInteractionStylePython-TestStyleTrackballCamera vtkInteractionStylePython-TestStyleTrackballCamera] | [http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkInteractionStylePython-TestStyleTrackballCamera vtkInteractionStylePython-TestStyleTrackballCamera] | ||
| 1 | | 1 | ||
| | | AIX [http://open.cdash.org/testDetails.php?test=178422784&build=2824162 Major regression failure] | ||
| Unassigned | | Unassigned | ||
|- | |- | ||
| Line 107: | Line 108: | ||
[http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling] | [http://open.cdash.org/testSummary.php?project=11&date=2013-02-23&name=vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling vtkRenderingOpenGLCxx-TestTextureRGBADepthPeeling] | ||
| 1 | | 1 | ||
| | | [http://open.cdash.org/testDetails.php?test=178390201&build=2823952 Intermittent regression failure] | ||
| Unassigned | | Unassigned | ||
|- | |- | ||
Revision as of 22:03, 23 February 2013
The VTK Nightly Expected builds show the status of VTK on VTK supported platforms at a fixed time each day. Although the Gerrit Review Process and Continuous Builds attempt to catch build defects early, it is the Nightly Expected builds that show the health of VTK on a broader collection of systems. Recurring nightly failures make it difficult for a developer to see if her changes have affected the spectrum of platforms.
This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" to resolve these issues.
The basic methodology (from Wikipedia) consists of the following five steps:
- Define process goals that are consistent with customer demands and VTK's strategy.
- Measure key aspects of the current process and collect relevant data.
- Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
- Improve or optimize the process.
- Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.
Define
Keep the number of VTK Nightly Expected recurring test failures(defects) to 0. When the defects are above 0, developers find it difficult to notice that their changes introduce new defects.
Measure
Recent summaries:
Analyze
| Test | Failing | Defect | Assigned To |
|
vtkFiltersParallelStatisticsCxx-MPI-TestRandomPMomentStatisticsMPI |
1 | Memory Leak | Unassigned |
| 1 | Failing since Feb 4 | Unassigned | |
| 1 | AIX only | Unassigned | |
| 1 | Small image regression | Unassigned | |
| 1 | AIX only Image regression | Unassigned | |
| 1 | Python syntax | Unassigned Existing gerrit topic | |
|
vtkRenderingVolumePython-TestFixedPointRayCasterNearestCropped |
1 | Image regression
AIX exit |
Unassigned |
| 1 | Failed to connect | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Major regression failure | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | AIX Major regression failure | Unassigned | |
| 1 | Intermittent regression failure | Unassigned | |
| 1 | Unknown | Unassigned | |
|
vtkRenderingVolumePython-TestFixedPointRayCasterLinearCropped |
1 | Unknown | Unassigned |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
| 1 | Unknown | Unassigned | |
|
vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DGeometryMaterial |
2 | Unknown | Unassigned |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
|
vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DUnstructured |
2 | Unknown | Unassigned |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
|
vtkFiltersHyperTreeCxx-TestHyperTreeGridTernarySphereMaterial |
2 | Unknown | Unassigned |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
|
vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DContourMaterial |
2 | Unknown | Unassigned |
|
vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DUnstructuredMaterial |
2 | Unknown | Unassigned |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
|
vtkFiltersHyperTreeCxx-TestHyperTreeGridTernary3DAxisCutMaterial |
2 | Unknown | Unassigned |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 2 | Unknown | Unassigned | |
| 3 | Unknown | Unassigned | |
| 3 | Unknown | Unassigned | |
|
vtkIOExportCxx-TestGL2PSExporterVolumeRaster-VerifyRasterizedPNG |
3 | Unknown | Unassigned |
| 4 | Unknown | Unassigned | |
| 5 | Timeout | Unassigned |
Improve
The following Gerrit reviews addressed the defects.
Control
Once reduced to 0, developer diligence is needed to keep the defects to 0.
