VTK/SoftwareQuality/CodeCoverage/Winter2014

From KitwarePublic
Jump to navigationJump to search

Testing is critical to high quality software. There are a number of aspects to the VTK's software quality. One important aspect is code coverage that measures how many lines of code are covered by the test suite.

This experiment seeks to increase VTK's code coverage.

This experiment uses the DMAIC methodology of the Six Sigma management process to "Define", "Measure", "Analyze", "Improve" and "Control" test code coverage in VTK.

The basic methodology (from Wikipedia) consists of the following five steps:

  • Define process goals that are consistent with customer demands and VTK's strategy.
  • Measure key aspects of the current process and collect relevant data.
  • Analyze the data to verify cause-and-effect relationships. Determine what the relationships are, and attempt to ensure that all factors have been considered.
  • Improve or optimize the process.
  • Control to ensure that any deviations from target are corrected before they result in defects. Set up pilot runs to establish software quality, move on to production, set up control mechanisms and continuously monitor the process.

Define

Increase the code coverage of vtk classes.

Measure

As of January 2014 the vtk tests covered 68% of vtk code.

Here is the current coverage

A summary of coverages by module

Analyze

Modules not shown on Karego-at's coverage

  • AcceleratorsDax - TOFIX
  • AcceleratorsPiston - TOFIX:
  • CommonColor - optional, turned on
  • CommonComputationalGeometry - OK, didn't show since coverage is high enough
  • FiltersParallelFlowPaths - optional, turned on
  • FiltersStatisticsGNUR - optional - TOFIX
  • GUISupportMFC - windows only, need a windows coverage machine to get it
  • GUISupportQtOpenGL - optional, turned on
  • GUISupportQtSQL - optional, turned on
  • GUISUpportQtWebkit - optional, turned on
  • InfovisBoost - optional, turned on
  • InfovisParallel - optional, turned on
  • IOSQL - OK, didn't show since coverage is high enough
  • IOVPIC - optional, turned on - but no tests - TOFIX
  • ViewsQt - optional, and only one test - TOFIX
  • WebCore - optional, turned on
  • WebWebGLExporter - optional, turned on
  • WebInstall, WebJavaScript, WebPython, WebApplications - optional, primarily not C++
  • (+ * TestingCore, Thirdparty*, Utilities* and Wrapping* which are expected)

Coverage by Module

Modules with < 70% are bold.

  • ChartsCore 70.06%
  • CommonCore 67.47%
  • CommonDataModel 71.67%
  • CommonExecutionModel 75.99%
  • CommonMath 77.97%
  • CommonMisc 59.87%
  • CommonSystem 40.27%
  • CommonTransforms 74.19%
  • DomainsChemistry 80.75%
  • FiltersAMR 70.15%
  • FiltersCore 73.09%
  • FiltersExtraction 66.13%
  • FiltersFlowPaths 67.21%
  • FiltersGeneral 71.94%
  • FiltersGeneric 73.87%
  • FiltersGeometry 78.51%
  • FiltersHybrid 72.44%
  • FiltersHyperTree 82.47%
  • FiltersImaging 23.58%
  • FiltersModeling 85.38%
  • FiltersParallel 86.72%
  • FiltersParallelGeometry 96.70%
  • FiltersParallelImaging 44.54%
  • FiltersParallelMPI 91.98%
  • FiltersParallelStatistics 78.11%
  • FiltersProgrammable 83.43%
  • FiltersReebGraph 64.69%
  • FiltersSelection 67.15%
  • FiltersSMP 66.96%
  • FiltersSources 85.73%
  • FiltersStatistics 78.92%
  • FiltersTexture 86.99%
  • FiltersVerdict 32.91%
  • GeovisCore 51.15%
  • GUISupportQt 20.59%
  • ImagingColor 78.00%
  • ImagingCore 81.98%
  • ImagingFourier 87.23%
  • ImagingGeneral 88.53%
  • ImagingHybrid 85.30%
  • ImagingMath 83.88%
  • ImagingMorphological 92.12%
  • ImagingSources 81.12%
  • ImagingStatistics 89.08%
  • ImagingStencil 88.96%
  • InfovisBoostGraphAlgorithms 75.62%
  • InfovisCore 63.98%
  • InfovisLayout 63.25%
  • InteractionImage 50.36%
  • InteractionStyle 59.08%
  • InteractionWidgets 63.80%
  • IOAMR 42.79%
  • IOCore 70.01%
  • IOEnSight 56.67%
  • IOExodus 39.86%
  • IOExport 64.75%
  • IOFFMPEG 72.49%
  • IOGDAL 83.33%
  • IOGeoJSON 83.47%
  • IOGeometry 41.97%
  • IOImage 59.92%
  • IOImport 64.60%
  • IOInfovis 79.99%
  • IOLegacy 55.64%
  • IOLSDyna 8.55%
  • IOMINC 76.37%
  • IOMovie 72.13%
  • IOMPIImage 49.34%
  • IOMPIParallel 9.09%
  • IOMySQL 18.02%
  • IONetCDF 38.85%
  • IOODBC 11.24%
  • IOParallel 35.47%
  • IOParallelExodus 14.80%
  • IOParallelLSDyna 38.20%
  • IOParallelNetCDF 19.03%
  • IOPLY 59.97%
  • IOPostgreSQL 16.58%
  • IOVideo 26.82%
  • IOXdmf2 54.18%
  • IOXML 72.98%
  • IOXMLParser 57.08%
  • ParallelCore 59.51%
  • ParallelMPI 64.58%
  • RenderingAnnotation 81.53%
  • RenderingContext2D 65.87%
  • RenderingCore 75.11%
  • RenderingFreeType 62.73%
  • RenderingFreeTypeFontConfig 70.48%
  • RenderingFreeTypeOpenGL 82.92%
  • RenderingGL2PS 65.16%
  • RenderingImage 85.59%
  • RenderingLabel 59.59%
  • RenderingLIC 81.59%
  • RenderingLOD 74.25%
  • RenderingMatplotlib 80.76%
  • RenderingOpenGL 56.65%
  • RenderingParallel 61.49%
  • RenderingParallelLIC 7.89%
  • RenderingQt 18.77%
  • RenderingTk 41.89%
  • RenderingVolume 85.46%
  • RenderingVolumeAMR 38.83%
  • RenderingVolumeOpenGL 68.88%
  • TestingGenericBridge 67.45%
  • TestingIOSQL 90.91%
  • TestingRendering 63.31%
  • ViewsContext2D 81.30%
  • ViewsCore 59.15%
  • ViewsGeovis 64.41%
  • ViewsInfovis 64.06%

Low hanging Fruit

Tests that were lost after modularization 

From: David Gobbi <david.gobbi@gmail.com>
Date: Sat, Mar 2, 2013 at 4:10 PM
Subject: Re: [vtk-developers] Coverage
To: VTK Developers <vtk-developers@vtk.org>
Here's a list of tests that are disabled in the master branch,
that were active in VTK 5.10.1.  It should give an idea of the
amount of coverage that has been lost since the last release.
  • ./Accelerators/Piston/Testing/Python/PistonTestCommon.py
  • ./Charts/Core/Testing/Cxx/TestGLSL.cxx
  • ./Common/Core/Testing/Cxx/TestArrayCasting.cxx
    • ./Common/Core/Testing/Cxx/CMakeLists.txt: # TestArrayCasting.cxx # Uses Boost in its own separate test.
  • ./Common/Core/Testing/Cxx/TestCxxFeatures.cxx
    • ./Common/Core/Testing/Cxx/CMakeLists.txt: # TestCxxFeatures.cxx # This is in its own exe too.
  • ./Common/Core/Testing/Cxx/TestInstantiator.cxx
    • ./Common/Core/Testing/Cxx/CMakeLists.txt: # TestInstantiator.cxx # Have not enabled instantiators.
  • ./Common/Core/Testing/Cxx/TestVariantArray.cxx
  • ./Common/Core/Testing/Cxx/CMakeLists.txt: #otherLookupTable.cxx
  • ./Common/Core/Testing/Cxx/CMakeLists.txt: #otherLookupTableWithEnabling.cxx
  • ./Common/DataModel/Testing/Cxx/CellLocator.cxx
  • ./Common/DataModel/Testing/Cxx/PointLocator.cxx
  • ./Common/DataModel/Testing/Cxx/TestGraph2.cxx
  • ./Common/DataModel/Testing/Cxx/TestGraphAttributes.cxx
  • ./Common/DataModel/Testing/Cxx/TestMeanValueCoordinatesInterpolation1.cxx
  • ./Common/DataModel/Testing/Cxx/TestMeanValueCoordinatesInterpolation2.cxx
  • ./Common/DataModel/Testing/Cxx/TestSmoothErrorMetric.cxx
  • ./Common/DataModel/Testing/Cxx/TestTable.cxx
  • ./Common/Transforms/Testing/Python/SliceOrder.py
  • ./Common/Transforms/Testing/Python/SliceOrder.py
  • ./Domains/Chemistry/Testing/Cxx/TestOpenQubeElectronicData.cxx
  • ./Domains/Chemistry/Testing/Cxx/TestOpenQubeMOPACDensity.cxx
  • ./Domains/Chemistry/Testing/Cxx/TestOpenQubeMOPACOrbital.cxx
  • ./Filters/Core/Testing/Cxx/TestNamedComponents.cxx
  • ./Filters/General/Testing/Cxx/ArrayExtractFactoredArray.cxx
  • ./Filters/General/Testing/Cxx/ArrayMatricizeArray.cxx
  • ./Filters/General/Testing/Cxx/ArrayNormalizeMatrixVectors.cxx
  • ./Filters/General/Testing/Cxx/CellTreeLocator.cxx
  • ./Filters/General/Testing/Cxx/TestPassArrays.cxx
  • ./Filters/General/Testing/Cxx/TestPassThrough.cxx
  • ./Filters/General/Testing/Cxx/TestTessellator.cxx
  • ./Filters/General/Testing/Cxx/expCos.cxx
  • ./Filters/Geometry/Testing/Cxx/TestRectilinearGridPartitioner.cxx
  • ./Filters/Geometry/Testing/Cxx/TestStructuredGridPartitioner.cxx
  • ./Filters/Hybrid/Testing/Cxx/TestBSplineWarp.cxx
    • ./Filters/Hybrid/Testing/Cxx/CMakeLists.txt: # TestBSplineWarp.cxx # Fixme after vtkImageViewer deps
  • ./Filters/Matlab/Testing/Cxx/TestMatlabEngineFilter.cxx
  • ./Filters/Matlab/Testing/Cxx/TestMatlabEngineInterface.cxx
  • ./Filters/ParallelStatistics/Testing/Cxx/TestRealDataPDescriptiveStatisticsMPI.cxx
  • ./Filters/Sources/Testing/Cxx/ArrayAPIDenseCoordinates.cxx
  • ./Filters/Sources/Testing/Cxx/ArrayDiagonalMatrixSource.cxx
  • ./Filters/Sources/Testing/Tcl/Disk.tcl
  • ./Filters/Sources/Testing/Tcl/OSCone.tcl
  • ./Filters/Sources/Testing/Tcl/TestButtonSource.tcl
  • ./Filters/Sources/Testing/Tcl/TestPlatonicSolids.tcl
  • ./Filters/Sources/Testing/Tcl/TestRegularPolygonSource.tcl
  • ./Filters/Sources/Testing/Tcl/coneResolution.tcl
  • ./Filters/Sources/Testing/Tcl/glyph2D.tcl
  • ./GUISupport/Qt/Testing/Python/TestConnection.py
  • ./GUISupport/Qt/Testing/Python/TestQVTKRenderWindowInteractor.py
  • ./GUISupport/Qt/Testing/Python/TestQVTKWidget.py
  • ./GUISupport/Qt/Testing/Python/TestQVTKWidget2.py
  • ./GUISupport/Qt/Testing/Python/TestTimeUtility.py
  • ./GUISupport/Qt/Testing/Python/TestvtkQtTableView.py
  • ./GUISupport/QtSQL/Testing/Cxx/TestQtSQLDatabase.cxx
  • ./IO/AMR/Testing/Cxx/TestAMRReadWrite.cxx
  • ./IO/AMR/Testing/Cxx/TestFlashReader.cxx
    • ./IO/AMR/Testing/Cxx/CMakeLists.txt: # TestFlashReader.cxx there is not data for this test
  • ./IO/Geometry/Testing/Cxx/TestAMRReadWrite.cxx
    • ./IO/Geometry/Testing/Cxx/CMakeLists.txt: TestAMRReadWrite.cxx,NO_VALID
  • ./IO/Geometry/Testing/Cxx/TestDataObjectIO.cxx
  • ./IO/Geometry/Testing/Cxx/TestIncrementalOctreePointLocator.cxx
  • ./IO/Geometry/Testing/Cxx/UnstructuredGridCellGradients.cxx
  • ./IO/Geometry/Testing/Cxx/UnstructuredGridFastGradients.cxx
  • ./IO/Geometry/Testing/Cxx/UnstructuredGridGradients.cxx
  • ./IO/Geometry/Testing/Python/TestPolygonWriters.py
    • ./IO/Geometry/Testing/Python/CMakeLists.txt:# vtk_add_test_python(TestPolygonWriters.py) # needs vtkCGMWriter (not built)
  • ./IO/Image/Testing/Cxx/TestImageReader2Factory.cxx
    • ./IO/Image/Testing/Cxx/CMakeLists.txt: # TestImageReader2Factory.cxx # fixme (deps not satisfied)
  • ./IO/Infovis/Testing/Cxx/TestDelimitedTextReader.cxx
  • ./IO/Infovis/Testing/Cxx/TestTulipReader.cxx
  • ./IO/Infovis/Testing/Cxx/TestTulipReaderClusters.cxx
  • ./IO/Movie/Testing/Cxx/TestMovieWriter.cxx
    • ./IO/Movie/Testing/Cxx/CMakeLists.txt: # TestMovieWriter.cxx # fixme (deps not satisfied)
  • ./IO/SQL/Testing/Cxx/TestSQLGraphReader.cxx
  • ./IO/XML/Testing/Tcl/TestXMLPolyDataIO.tcl
  • ./IO/XML/Testing/Tcl/TestXMLUnstructuredGridIO.tcl
  • ./Imaging/Core/Testing/Python/WindowLevelInterface.py
  • ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageAutoRange.cxx
  • ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageHistogram.cxx
  • ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageHistogramStatistics.cxx,NO_VALID
  • ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageResize.cxx
  • ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageResize3D.cxx
  • ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImageResizeCropping.cxx
  • ./Imaging/Core/Testing/Cxx/CMakeLists.txt: ImportExport.cxx,NO_VALID # todo (unsatistfied deps)
  • ./Imaging/Core/Testing/Cxx/ImageAutoRange.cxx
  • ./Imaging/Core/Testing/Cxx/ImageHistogram.cxx
  • ./Imaging/Core/Testing/Cxx/ImageHistogramStatistics.cxx
  • ./Imaging/Core/Testing/Cxx/ImageResize.cxx
  • ./Imaging/Core/Testing/Cxx/ImageResize3D.cxx
  • ./Imaging/Core/Testing/Cxx/ImageResizeCropping.cxx
  • ./Imaging/Core/Testing/Cxx/ImportExport.cxx
    • ./Imaging/Core/Testing/Python/CMakeLists.txt:# vtk_add_test_python(WindowLevelInterface.py) # TCL test not being made as well !!!
  • ./Infovis/BoostGraphAlgorithms/Testing/Cxx/BoostArrayLogWeighting.cxx
    • ./Infovis/BoostGraphAlgorithms/Testing/Cxx/CMakeLists.txt: # BoostArrayLogWeighting.cxx # todo (unsatisfied deps)
  • ./Infovis/BoostGraphAlgorithms/Testing/Cxx/TestBoostBrandesCentrality.cxx
    • ./Infovis/BoostGraphAlgorithms/Testing/Cxx/CMakeLists.txt: # TestBoostBrandesCentrality.cxx # todo (unsatisfied deps)
  • ./Infovis/Core/Testing/Cxx/ArrayDotProductSimilarity.cxx
  • ./Infovis/Core/Testing/Cxx/TestArrayNorm.cxx
    • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: # TestArrayNorm.cxx # todo (unsatisfied deps)
  • ./Infovis/Core/Testing/Cxx/TestConvexHull2D.cxx
    • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: # TestConvexHull2D.cxx # todo (unsatisfied deps)
  • ./Infovis/Core/Testing/Cxx/TestKCoreDecomposition.cxx
  • ./Infovis/Core/Testing/Cxx/TestPruneTreeFilter.cxx
    • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: TestPruneTreeFilter.cxx
  • ./Infovis/Core/Testing/Cxx/TestStringToCategory.cxx
  • ./Infovis/Core/Testing/Cxx/TestTableToGraph.cxx
  • ./Infovis/Core/Testing/Cxx/TestTimePoint.cxx
    • ./Infovis/Core/Testing/Cxx/CMakeLists.txt: # TestTimePoint.cxx # deprecated dep
  • ./Infovis/Layout/Testing/Cxx/TestArcEdges.cxx
  • ./Infovis/Layout/Testing/Cxx/TestAssignCoordinates.cxx
    • ./Infovis/Layout/Testing/Cxx/CMakeLists.txt: # TestAssignCoordinates.cxx # todo (unsatisfied deps)
  • ./Infovis/Layout/Testing/Cxx/TestChacoGraphReader.cxx
    • ./Infovis/Layout/Testing/Cxx/CMakeLists.txt: TestChacoGraphReader.cxx
  • ./Infovis/Layout/Testing/Cxx/TestCosmicTreeLayoutStrategy.cxx
    • ./Infovis/Layout/Testing/Cxx/CMakeLists.txt: # TestCosmicTreeLayoutStrategy.cxx # todo (unsatisfied deps)
  • ./Infovis/Layout/Testing/Cxx/TestGroupLeafVertices.cxx
  • ./Infovis/Layout/Testing/Cxx/TestKCoreLayout.cxx
  • ./Infovis/Layout/Testing/Cxx/TestKdTreeBoxSelection.cxx
  • ./Infovis/Layout/Testing/Cxx/TestPruneTreeFilter.cxx
  • ./Infovis/Layout/Testing/Cxx/TestSimple3DCirclesStrategy.cxx
  • ./Interaction/Style/Testing/Python/TestStyleBase.py
  • ./Interaction/Widgets/Testing/Cxx/ImagePlaneWidget.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # ImagePlaneWidget.cxx # todo (unsatisfied deps)
  • ./Interaction/Widgets/Testing/Cxx/TestBrokenLineWidget.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestBrokenLineWidget.cxx # todo (unsatisfied deps)
  • ./Interaction/Widgets/Testing/Cxx/TestButtonWidget.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestButtonWidget.cxx # todo (unsatistfied deps)
  • ./Interaction/Widgets/Testing/Cxx/TestImageActorContourWidget.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestImageActorContourWidget.cxx # todo(unsatistfied deps)
  • ./Interaction/Widgets/Testing/Cxx/TestOrthoPlanes.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestOrthoPlanes.cxx # todo (unsatistfied deps)
  • ./Interaction/Widgets/Testing/Cxx/TestResliceCursorWidget2.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestResliceCursorWidget2.cxx # todo (unsatistfied deps)
  • ./Interaction/Widgets/Testing/Cxx/TestResliceCursorWidget3.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestResliceCursorWidget3.cxx # todo (unsatistfied deps)
  • ./Interaction/Widgets/Testing/Cxx/TestSplineWidget.cxx
    • ./Interaction/Widgets/Testing/Cxx/CMakeLists.txt: # TestSplineWidget.cxx # todo (unsatistfied deps)
  • ./Interaction/Widgets/Testing/Cxx/vtkImplicitPlaneWidget2Test1.cxx
  • ./Parallel/Core/Testing/Python/DummyController.cxx
  • ./Parallel/Core/Testing/Python/SocketClient.cxx
  • ./Parallel/Core/Testing/Python/SocketServer.cxx
  • ./Parallel/Core/Testing/Python/TestSockets.py
  • ./Rendering/Annotation/Testing/Cxx/TestAxisActor3D.cxx
    • ./Rendering/Annotation/Testing/Cxx/CMakeLists.txt: TestAxisActor3D.cxx
  • ./Rendering/Core/Testing/Cxx/FrustumClip.cxx
  • ./Rendering/Core/Testing/Cxx/Mace.cxx
  • ./Rendering/Core/Testing/Cxx/RGrid.cxx
  • ./Rendering/Core/Testing/Cxx/TestActor2DTextures.cxx
  • ./Rendering/Core/Testing/Cxx/TestBareScalarsToColors.cxx
  • ./Rendering/Core/Testing/Cxx/TestHierarchicalBoxPipeline.cxx
    • ./Rendering/Core/Testing/Cxx/CMakeLists.txt: # TestHierarchicalBoxPipeline.cxx # legacy code
  • ./Rendering/Core/Testing/Cxx/TestMapVectorsAsRGBColors.cxx
  • ./Rendering/Core/Testing/Cxx/TestMapVectorsToColors.cxx
  • ./Rendering/Core/Testing/Cxx/TestTDxGeo.cxx
  • ./Rendering/Core/Testing/Cxx/TestViewDependentErrorMetric.cxx
  • ./Rendering/Core/Testing/Cxx/otherLookupTable.cxx
    • ./Rendering/Core/Testing/Cxx/CMakeLists.txt: # otherLookupTable.cxx # fixme
  • ./Rendering/Core/Testing/Cxx/otherLookupTableWithEnabling.cxx
    • ./Rendering/Core/Testing/Cxx/CMakeLists.txt: # otherLookupTableWithEnabling.cxx # fixme
  • ./Rendering/Core/Testing/Python/TestParallelCoordinates.py
    • ./Rendering/Core/Testing/Python/CMakeLists.txt:# vtk_add_test_python(TestParallelCoordinates.py) # disabled in TCL too
  • ./Rendering/Core/Testing/Tcl/TestCgShader.tcl
  • ./Rendering/Core/Testing/Tcl/TestGLSLShader.tcl
  • ./Rendering/Core/Testing/Tcl/TestGenericVertexAttributesCg.tcl
  • ./Rendering/Core/Testing/Tcl/TestGenericVertexAttributesGLSL.tcl
  • ./Rendering/Core/Testing/Tcl/TestParallelCoordinates.tcl
  • ./Rendering/Qt/Testing/Cxx/TestQtInitialization.cxx
  • ./Rendering/Volume/Testing/Python/cursor3D.py
  • ./Rendering/Volume/Testing/Python/CMakeLists.txt:vtk_add_test_python(cursor3D.py)
  • ./Views/Geovis/Testing/Cxx/TestGeoView.cxx
  • ./Views/Geovis/Testing/Cxx/TestGeoView2D.cxx
  • ./Views/Infovis/Testing/Cxx/TestQtTreeRingLabeler.cxx
  • ./Views/Qt/Testing/Cxx/TestVtkQtTableView.cxx

Improve

We should document these much better.

  • Cookie cutter approach to add a test is …
  • Useful tools to call from your test are ...

Control

This is a useful script for computing code coverage. Currently only works on linux.

Retrieved from "https://public.kitware.com/Wiki/index.php?title=VTK/SoftwareQuality/CodeCoverage/Winter2014&oldid=55109"