VTK/SoftwareQuality/CodeCoverage/Winter2014

Testing is critical to high quality software. There are a number of aspects to the VTK's software quality. One important aspect is code coverage that measures how many lines of code are covered by the test suite.

This experiment seeks to increase VTK's code coverage.

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" test code coverage in VTK.

The basic methodology (from Wikipedia) consists of the following five steps:

• Define process goals that are consistent with customer demands and VTK's strategy.
• Measure key aspects of the current process and collect relevant data.
• Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
• Improve or optimize the process.
• Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Increase the code coverage of vtk classes.

Measure

As of January 2014 the vtk tests covered 67% of vtk code.

Analyze

From: David Gobbi <david.gobbi@gmail.com>
Date: Sat, Mar 2, 2013 at 4:10 PM
Subject: Re: [vtk-developers] Coverage
To: VTK Developers <vtk-developers@vtk.org>
Here's a list of tests that are disabled in the master branch,
that were active in VTK 5.10.1.  It should give an idea of the
amount of coverage that has been lost since the last release.

• ./Accelerators/Piston/Testing/Python/PistonTestCommon.py
• ./Charts/Core/Testing/Cxx/TestGLSL.cxx
• ./Common/Core/Testing/Cxx/TestArrayCasting.cxx
  • ./Common/Core/Testing/Cxx/CMakeLists.txt: # TestArrayCasting.cxx # Uses Boost in its own separate test.
• ./Common/Core/Testing/Cxx/TestCxxFeatures.cxx
  • ./Common/Core/Testing/Cxx/CMakeLists.txt: # TestCxxFeatures.cxx # This is in its own exe too.
• ./Common/Core/Testing/Cxx/TestInstantiator.cxx
  • ./Common/Core/Testing/Cxx/CMakeLists.txt: # TestInstantiator.cxx # Have not enabled instantiators.
• ./Common/Core/Testing/Cxx/TestVariantArray.cxx
• ./Common/Core/Testing/Cxx/CMakeLists.txt: #otherLookupTable.cxx
• ./Common/Core/Testing/Cxx/CMakeLists.txt: #otherLookupTableWithEnabling.cxx
• ./Common/DataModel/Testing/Cxx/CellLocator.cxx
• ./Common/DataModel/Testing/Cxx/PointLocator.cxx
• ./Common/DataModel/Testing/Cxx/TestGraph2.cxx
• ./Common/DataModel/Testing/Cxx/TestGraphAttributes.cxx
• ./Common/DataModel/Testing/Cxx/TestMeanValueCoordinatesInterpolation1.cxx
• ./Common/DataModel/Testing/Cxx/TestMeanValueCoordinatesInterpolation2.cxx
• ./Common/DataModel/Testing/Cxx/TestSmoothErrorMetric.cxx
• ./Common/DataModel/Testing/Cxx/TestTable.cxx
• ./Common/Transforms/Testing/Python/SliceOrder.py
• ./Common/Transforms/Testing/Python/SliceOrder.py
• ./Domains/Chemistry/Testing/Cxx/TestOpenQubeElectronicData.cxx
• ./Domains/Chemistry/Testing/Cxx/TestOpenQubeMOPACDensity.cxx
• ./Domains/Chemistry/Testing/Cxx/TestOpenQubeMOPACOrbital.cxx
• ./Filters/Core/Testing/Cxx/TestNamedComponents.cxx
• ./Filters/General/Testing/Cxx/ArrayExtractFactoredArray.cxx
• ./Filters/General/Testing/Cxx/ArrayMatricizeArray.cxx
• ./Filters/General/Testing/Cxx/ArrayNormalizeMatrixVectors.cxx
• ./Filters/General/Testing/Cxx/CellTreeLocator.cxx
• ./Filters/General/Testing/Cxx/TestPassArrays.cxx
• ./Filters/General/Testing/Cxx/TestPassThrough.cxx
• ./Filters/General/Testing/Cxx/TestTessellator.cxx
• ./Filters/General/Testing/Cxx/expCos.cxx
• ./Filters/Geometry/Testing/Cxx/TestRectilinearGridPartitioner.cxx
• ./Filters/Geometry/Testing/Cxx/TestStructuredGridPartitioner.cxx
• ./Filters/Hybrid/Testing/Cxx/TestBSplineWarp.cxx
  • ./Filters/Hybrid/Testing/Cxx/CMakeLists.txt: # TestBSplineWarp.cxx # Fixme after vtkImageViewer deps
• ./Filters/Matlab/Testing/Cxx/TestMatlabEngineFilter.cxx
• ./Filters/Matlab/Testing/Cxx/TestMatlabEngineInterface.cxx
• ./Filters/ParallelStatistics/Testing/Cxx/TestRealDataPDescriptiveStatisticsMPI.cxx
• ./Filters/Sources/Testing/Cxx/ArrayAPIDenseCoordinates.cxx
• ./Filters/Sources/Testing/Cxx/ArrayDiagonalMatrixSource.cxx
• ./Filters/Sources/Testing/Tcl/Disk.tcl
• ./Filters/Sources/Testing/Tcl/OSCone.tcl
• ./Filters/Sources/Testing/Tcl/TestButtonSource.tcl
• ./Filters/Sources/Testing/Tcl/TestPlatonicSolids.tcl
• ./Filters/Sources/Testing/Tcl/TestRegularPolygonSource.tcl
• ./Filters/Sources/Testing/Tcl/coneResolution.tcl
• ./Filters/Sources/Testing/Tcl/glyph2D.tcl
• ./GUISupport/Qt/Testing/Python/TestConnection.py
• ./GUISupport/Qt/Testing/Python/TestQVTKRenderWindowInteractor.py
• ./GUISupport/Qt/Testing/Python/TestQVTKWidget.py
• ./GUISupport/Qt/Testing/Python/TestQVTKWidget2.py
• ./GUISupport/Qt/Testing/Python/TestTimeUtility.py
• ./GUISupport/Qt/Testing/Python/TestvtkQtTableView.py
• ./GUISupport/QtSQL/Testing/Cxx/TestQtSQLDatabase.cxx
• ./IO/AMR/Testing/Cxx/TestAMRReadWrite.cxx
• ./IO/AMR/Testing/Cxx/TestFlashReader.cxx
  • ./IO/AMR/Testing/Cxx/CMakeLists.txt: # TestFlashReader.cxx there is not data for this test
• ./IO/Geometry/Testing/Cxx/TestAMRReadWrite.cxx
  • ./IO/Geometry/Testing/Cxx/CMakeLists.txt: TestAMRReadWrite.cxx,NO_VALID
• ./IO/Geometry/Testing/Cxx/TestDataObjectIO.cxx
• ./IO/Geometry/Testing/Cxx/TestIncrementalOctreePointLocator.cxx
• ./IO/Geometry/Testing/Cxx/UnstructuredGridCellGradients.cxx
• ./IO/Geometry/Testing/Cxx/UnstructuredGridFastGradients.cxx
• ./IO/Geometry/Testing/Cxx/UnstructuredGridGradients.cxx
• ./IO/Geometry/Testing/Python/TestPolygonWriters.py
  • ./IO/Geometry/Testing/Python/CMakeLists.txt:# vtk_add_test_python(TestPolygonWriters.py) # needs vtkCGMWriter (not built)
• ./IO/Image/Testing/Cxx/TestImageReader2Factory.cxx
  • ./IO/Image/Testing/Cxx/CMakeLists.txt: # TestImageReader2Factory.cxx # fixme (deps not satisfied)
• ./IO/Infovis/Testing/Cxx/TestDelimitedTextReader.cxx
• ./IO/Infovis/Testing/Cxx/TestTulipReader.cxx
• ./IO/Infovis/Testing/Cxx/TestTulipReaderClusters.cxx
• ./IO/Movie/Testing/Cxx/TestMovieWriter.cxx
  • ./IO/Movie/Testing/Cxx/CMakeLists.txt: # TestMovieWriter.cxx # fixme (deps not satisfied)
• ./IO/SQL/Testing/Cxx/TestSQLGraphReader.cxx
• ./IO/XML/Testing/Tcl/TestXMLPolyDataIO.tcl
• ./IO/XML/Testing/Tcl/TestXMLUnstructuredGridIO.tcl
• ./Imaging/Core/Testing/Python/WindowLevelInterface.py
• ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageAutoRange.cxx
• ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageHistogram.cxx
• ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageHistogramStatistics.cxx,NO_VALID
• ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageResize.cxx
• ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageResize3D.cxx
• ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageResizeCropping.cxx
• ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImportExport.cxx,NO_VALID # todo (unsatistfied deps)
• ./Imaging/Core/Testing/Cxx/ImageAutoRange.cxx
• ./Imaging/Core/Testing/Cxx/ImageHistogram.cxx
• ./Imaging/Core/Testing/Cxx/ImageHistogramStatistics.cxx
• ./Imaging/Core/Testing/Cxx/ImageResize.cxx
• ./Imaging/Core/Testing/Cxx/ImageResize3D.cxx
• ./Imaging/Core/Testing/Cxx/ImageResizeCropping.cxx
• ./Imaging/Core/Testing/Cxx/ImportExport.cxx
  • ./Imaging/Core/Testing/Python/CMakeLists.txt:# vtk_add_test_python(WindowLevelInterface.py) # TCL test not being made as well !!!
• ./Infovis/BoostGraphAlgorithms/Testing/Cxx/BoostArrayLogWeighting.cxx
  • ./Infovis/BoostGraphAlgorithms/Testing/Cxx/CMakeLists.txt: # BoostArrayLogWeighting.cxx # todo (unsatisfied deps)
• ./Infovis/BoostGraphAlgorithms/Testing/Cxx/TestBoostBrandesCentrality.cxx
  • ./Infovis/BoostGraphAlgorithms/Testing/Cxx/CMakeLists.txt: # TestBoostBrandesCentrality.cxx # todo (unsatisfied deps)
• ./Infovis/Core/Testing/Cxx/ArrayDotProductSimilarity.cxx
• ./Infovis/Core/Testing/Cxx/TestArrayNorm.cxx
  • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: # TestArrayNorm.cxx # todo (unsatisfied deps)
• ./Infovis/Core/Testing/Cxx/TestConvexHull2D.cxx
  • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: # TestConvexHull2D.cxx # todo (unsatisfied deps)
• ./Infovis/Core/Testing/Cxx/TestKCoreDecomposition.cxx
• ./Infovis/Core/Testing/Cxx/TestPruneTreeFilter.cxx
  • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: TestPruneTreeFilter.cxx
• ./Infovis/Core/Testing/Cxx/TestStringToCategory.cxx
• ./Infovis/Core/Testing/Cxx/TestTableToGraph.cxx
• ./Infovis/Core/Testing/Cxx/TestTimePoint.cxx
  • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: # TestTimePoint.cxx # deprecated dep
• ./Infovis/Layout/Testing/Cxx/TestArcEdges.cxx
• ./Infovis/Layout/Testing/Cxx/TestAssignCoordinates.cxx
  • ./Infovis/Layout/Testing/Cxx/CMakeLists.txt: # TestAssignCoordinates.cxx # todo (unsatisfied deps)
• ./Infovis/Layout/Testing/Cxx/TestChacoGraphReader.cxx
  • ./Infovis/Layout/Testing/Cxx/CMakeLists.txt: TestChacoGraphReader.cxx
• ./Infovis/Layout/Testing/Cxx/TestCosmicTreeLayoutStrategy.cxx
  • ./Infovis/Layout/Testing/Cxx/CMakeLists.txt: # TestCosmicTreeLayoutStrategy.cxx # todo (unsatisfied deps)
• ./Infovis/Layout/Testing/Cxx/TestGroupLeafVertices.cxx
• ./Infovis/Layout/Testing/Cxx/TestKCoreLayout.cxx
• ./Infovis/Layout/Testing/Cxx/TestKdTreeBoxSelection.cxx
• ./Infovis/Layout/Testing/Cxx/TestPruneTreeFilter.cxx
• ./Infovis/Layout/Testing/Cxx/TestSimple3DCirclesStrategy.cxx
• ./Interaction/Style/Testing/Python/TestStyleBase.py
• ./Interaction/Widgets/Testing/Cxx/ImagePlaneWidget.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # ImagePlaneWidget.cxx # todo (unsatisfied deps)
• ./Interaction/Widgets/Testing/Cxx/TestBrokenLineWidget.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestBrokenLineWidget.cxx # todo (unsatisfied deps)
• ./Interaction/Widgets/Testing/Cxx/TestButtonWidget.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestButtonWidget.cxx # todo (unsatistfied deps)
• ./Interaction/Widgets/Testing/Cxx/TestImageActorContourWidget.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestImageActorContourWidget.cxx # todo(unsatistfied deps)
• ./Interaction/Widgets/Testing/Cxx/TestOrthoPlanes.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestOrthoPlanes.cxx # todo (unsatistfied deps)
• ./Interaction/Widgets/Testing/Cxx/TestResliceCursorWidget2.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestResliceCursorWidget2.cxx # todo (unsatistfied deps)
• ./Interaction/Widgets/Testing/Cxx/TestResliceCursorWidget3.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestResliceCursorWidget3.cxx # todo (unsatistfied deps)
• ./Interaction/Widgets/Testing/Cxx/TestSplineWidget.cxx
  • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestSplineWidget.cxx # todo (unsatistfied deps)
• ./Interaction/Widgets/Testing/Cxx/vtkImplicitPlaneWidget2Test1.cxx
• ./Parallel/Core/Testing/Python/DummyController.cxx
• ./Parallel/Core/Testing/Python/SocketClient.cxx
• ./Parallel/Core/Testing/Python/SocketServer.cxx
• ./Parallel/Core/Testing/Python/TestSockets.py
• ./Rendering/Annotation/Testing/Cxx/TestAxisActor3D.cxx
  • ./Rendering/Annotation/Testing/Cxx/CMakeLists.txt: TestAxisActor3D.cxx
• ./Rendering/Core/Testing/Cxx/FrustumClip.cxx
• ./Rendering/Core/Testing/Cxx/Mace.cxx
• ./Rendering/Core/Testing/Cxx/RGrid.cxx
• ./Rendering/Core/Testing/Cxx/TestActor2DTextures.cxx
• ./Rendering/Core/Testing/Cxx/TestBareScalarsToColors.cxx
• ./Rendering/Core/Testing/Cxx/TestHierarchicalBoxPipeline.cxx
  • ./Rendering/Core/Testing/Cxx/CMakeLists.txt: # TestHierarchicalBoxPipeline.cxx # legacy code
• ./Rendering/Core/Testing/Cxx/TestMapVectorsAsRGBColors.cxx
• ./Rendering/Core/Testing/Cxx/TestMapVectorsToColors.cxx
• ./Rendering/Core/Testing/Cxx/TestTDxGeo.cxx
• ./Rendering/Core/Testing/Cxx/TestViewDependentErrorMetric.cxx
• ./Rendering/Core/Testing/Cxx/otherLookupTable.cxx
  • ./Rendering/Core/Testing/Cxx/CMakeLists.txt: # otherLookupTable.cxx # fixme
• ./Rendering/Core/Testing/Cxx/otherLookupTableWithEnabling.cxx
  • ./Rendering/Core/Testing/Cxx/CMakeLists.txt: # otherLookupTableWithEnabling.cxx # fixme
• ./Rendering/Core/Testing/Python/TestParallelCoordinates.py
  • ./Rendering/Core/Testing/Python/CMakeLists.txt:# vtk_add_test_python(TestParallelCoordinates.py) # disabled in TCL too
• ./Rendering/Core/Testing/Tcl/TestCgShader.tcl
• ./Rendering/Core/Testing/Tcl/TestGLSLShader.tcl
• ./Rendering/Core/Testing/Tcl/TestGenericVertexAttributesCg.tcl
• ./Rendering/Core/Testing/Tcl/TestGenericVertexAttributesGLSL.tcl
• ./Rendering/Core/Testing/Tcl/TestParallelCoordinates.tcl
• ./Rendering/Qt/Testing/Cxx/TestQtInitialization.cxx
• ./Rendering/Volume/Testing/Python/cursor3D.py
• ./Rendering/Volume/Testing/Python/CMakeLists.txt:vtk_add_test_python(cursor3D.py)
• ./Views/Geovis/Testing/Cxx/TestGeoView.cxx
• ./Views/Geovis/Testing/Cxx/TestGeoView2D.cxx
• ./Views/Infovis/Testing/Cxx/TestQtTreeRingLabeler.cxx
• ./Views/Qt/Testing/Cxx/TestVtkQtTableView.cxx