CTest/Coverage: Difference between revisions

From KitwarePublic
Jump to navigationJump to search
No edit summary
(Add explicit preformat markup)
Line 4: Line 4:
with debug symbols, without optimization, and with special flags. These flags are:
with debug symbols, without optimization, and with special flags. These flags are:


<pre>
  -fprofile-arcs -ftest-coverage
  -fprofile-arcs -ftest-coverage
</pre>


Also make sure to pass these flags to C compiler, CXX compiler, and the linker. For example:
Also make sure to pass these flags to C compiler, CXX compiler, and the linker. For example:


<pre>
  CXXFLAGS="-g -O0 -Wall -W -Wshadow -Wunused-variable \
  CXXFLAGS="-g -O0 -Wall -W -Wshadow -Wunused-variable \
   -Wunused-parameter -Wunused-function -Wunused -Wno-system-headers \
   -Wunused-parameter -Wunused-function -Wunused -Wno-system-headers \
Line 13: Line 16:
  CFLAGS="-g -O0 -Wall -W -fprofile-arcs -ftest-coverage"
  CFLAGS="-g -O0 -Wall -W -fprofile-arcs -ftest-coverage"
  LDFLAGS="-fprofile-arcs -ftest-coverage"
  LDFLAGS="-fprofile-arcs -ftest-coverage"
</pre>


Note: Older versions of ccache will cause the coverage test to fail. Ctest will report that it cannot find any coverage files and this occurs because ccache loses them in the two step compile-link process. Newer versions have a patch to prevent this (e.g. gentoo ebuild 2.4-rt8+).
Note: Older versions of ccache will cause the coverage test to fail. Ctest will report that it cannot find any coverage files and this occurs because ccache loses them in the two step compile-link process. Newer versions have a patch to prevent this (e.g. gentoo ebuild 2.4-rt8+).
Line 56: Line 60:
Here is an example ctest -S script that runs a CMake coverage dashboard with bullseye:
Here is an example ctest -S script that runs a CMake coverage dashboard with bullseye:


<pre>
  CMAKE_MINIMUM_REQUIRED (VERSION 2.2)
  CMAKE_MINIMUM_REQUIRED (VERSION 2.2)
   
   
  # put bullseye path first
  # put bullseye path first
  SET(ENV{PATH} "c:/Program\ Files/BullseyeCoverage/bin/:${PATH}")
  SET(ENV{PATH} "c:/Program\ Files/BullseyeCoverage/bin/:${PATH}")
 
  # set variables for script
  # set variables for script
  SET (CTEST_SITE          "discworld.kitware")
  SET (CTEST_SITE          "discworld.kitware")
Line 74: Line 79:
  SET (CTEST_CMAKE_GENERATOR "NMake Makefiles")
  SET (CTEST_CMAKE_GENERATOR "NMake Makefiles")
  SET (CTEST_PROJECT_NAME "CMake")
  SET (CTEST_PROJECT_NAME "CMake")
 
  # clear the binary directory and create an initial cache
  # clear the binary directory and create an initial cache
  CTEST_EMPTY_BINARY_DIRECTORY (${CTEST_BINARY_DIRECTORY})
  CTEST_EMPTY_BINARY_DIRECTORY (${CTEST_BINARY_DIRECTORY})
Line 82: Line 87:
  CTEST_TEST_CTEST:BOOL=1
  CTEST_TEST_CTEST:BOOL=1
  ")
  ")
 
  # Find cov01, the bullseye tool to turn on/off coverage
  # Find cov01, the bullseye tool to turn on/off coverage
  FIND_PROGRAM(COV01 cov01)
  FIND_PROGRAM(COV01 cov01)
Line 94: Line 99:
   MESSAGE(FATAL_ERROR "could not run cov01 -0")
   MESSAGE(FATAL_ERROR "could not run cov01 -0")
  ENDIF(RES)
  ENDIF(RES)
 
  # do the dashboard steps
  # do the dashboard steps
  CTEST_START (NightlyCoverage)
  CTEST_START (NightlyCoverage)
Line 110: Line 115:
  CTEST_SUBMIT ()
  CTEST_SUBMIT ()
  EXECUTE_PROCESS(COMMAND ${COV01} -0 RESULT_VARIABLE RES)
  EXECUTE_PROCESS(COMMAND ${COV01} -0 RESULT_VARIABLE RES)
</pre>


==Coverage With Python==
==Coverage With Python==
Line 115: Line 121:
===Simple case ===
===Simple case ===


<pre>
$ cat hello.py
$ cat hello.py
<pre>
"""
"""
Comment #1
Comment #1
Line 140: Line 146:
will produce a file  
will produce a file  


<pre>
  hello.cover
  hello.cover
</pre>


<pre>
  $ cat hello.cover
  $ cat hello.cover
<pre>
       """
       """
       Comment #1
       Comment #1
Line 149: Line 157:
     1: def untested():
     1: def untested():
         print "untested"
         print "untested"
 
       """
       """
       Comment #2
       Comment #2
Line 155: Line 163:
     1: def tested():
     1: def tested():
     1:  print "tested"
     1:  print "tested"
 
       """
       """
       Comment #3
       Comment #3
Line 167: Line 175:
Using python code from VTK:
Using python code from VTK:


<pre>
  VTK/Wrapping/Python/vtk/test/Testing.py
  VTK/Wrapping/Python/vtk/test/Testing.py
</pre>


$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 Testing.py
$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 Testing.py
Line 173: Line 183:
will produce:
will produce:


<pre>
  ls *.cover
  ls *.cover
  Testing.cover      vtk.__init__.cover  vtk.filtering.cover        vtk.graphics.cover  vtk.imaging.cover  vtk.io.cover        vtk.rendering.cover      vtk.util.__init__.cover      vtk.util.vtkMethodParser.cover  vtk.widgets.cover
  Testing.cover      vtk.__init__.cover  vtk.filtering.cover        vtk.graphics.cover  vtk.imaging.cover  vtk.io.cover        vtk.rendering.cover      vtk.util.__init__.cover      vtk.util.vtkMethodParser.cover  vtk.widgets.cover
  vtk.__helper.cover  vtk.common.cover    vtk.genericfiltering.cover  vtk.hybrid.cover    vtk.infovis.cover  vtk.parallel.cover  vtk.test.BlackBox.cover  vtk.util.vtkConstants.cover  vtk.volumerendering.cover
  vtk.__helper.cover  vtk.common.cover    vtk.genericfiltering.cover  vtk.hybrid.cover    vtk.infovis.cover  vtk.parallel.cover  vtk.test.BlackBox.cover  vtk.util.vtkConstants.cover  vtk.volumerendering.cover
</pre>


Example:
Example:

Revision as of 18:33, 24 April 2018

Coverage With C++

Currently coverage is only supported on gcc compiler. To perform coverage test, make sure that your code is built with debug symbols, without optimization, and with special flags. These flags are:

 -fprofile-arcs -ftest-coverage

Also make sure to pass these flags to C compiler, CXX compiler, and the linker. For example:

 CXXFLAGS="-g -O0 -Wall -W -Wshadow -Wunused-variable \
  -Wunused-parameter -Wunused-function -Wunused -Wno-system-headers \
  -Wno-deprecated -Woverloaded-virtual -Wwrite-strings -fprofile-arcs -ftest-coverage"
 CFLAGS="-g -O0 -Wall -W -fprofile-arcs -ftest-coverage"
 LDFLAGS="-fprofile-arcs -ftest-coverage"

Note: Older versions of ccache will cause the coverage test to fail. Ctest will report that it cannot find any coverage files and this occurs because ccache loses them in the two step compile-link process. Newer versions have a patch to prevent this (e.g. gentoo ebuild 2.4-rt8+).

To actually run the coverage testing, you need to have tests added with add_test in your CMakeLists.txt file, then you need to run a ctest -S script that calls ctest_coverage, or use one of the predefined coverage dashboard targets. (Use 'make help | grep Coverage' to see those.)

Example of running coverage analysis

Shared libs

You might have issues with shared libs, as reported here:

(I was not able to use that trick, on a debian/oldstable and gcc 3.3.5, ref here).

Other refs found on the net:

Conclusion: You should use gcc 3.4 and up, since gcc-3.3.5 and before have this issue.

C/C++ Coverage with Bullseye

As of June 28th 2007 CVS CMake, there is support for the cross platform coverage tool Bullseye in CMake. Bullseye is a cross platform commercial tool for coverage analysis. Information about Bullseye can be found here: http://www.bullseye.com/. You will need a license for Bullseye to run it.

Since Bullseye is a branch coverage tool, the output does not exacly match what Dart is expecting. So, for now it makes up a fake number for lines of code covered. The number is 100 * the number of functions in the file. (assume 100 lines per function). The percent coverage is a mix of branches covered and functions covered.

To use Bullseye with CTest, it is easiest to use the new format ctest scripts. You need to make sure the Bullseye installation bin directory is in your PATH first. This can be done by setting the PATH environment variable in the ctest script. Bullseye has a tool cov01 that can turn on/off the coverage tool. You run that to enable/disable the coverage. It should be disabled for the configure part of a project so that coverage is not done on the try/compile stuff. The environment variable COVFILE points to the coverage database file used by Bullseye to store the coverage information. This needs to be set in the ctest script.

Here is an example ctest -S script that runs a CMake coverage dashboard with bullseye:

 CMAKE_MINIMUM_REQUIRED (VERSION 2.2)
 
 # put bullseye path first
 SET(ENV{PATH} "c:/Program\ Files/BullseyeCoverage/bin/:${PATH}")
 
 # set variables for script
 SET (CTEST_SITE          "discworld.kitware")
 SET (CTEST_BUILD_NAME    "Windows")
 SET (CTEST_NOTES_FILES    
     "${CTEST_SCRIPT_DIRECTORY}/${CTEST_SCRIPT_NAME}")
 SET (CTEST_DASHBOARD_ROOT   "c:/Hoffman/My Builds/")
 SET (CTEST_SOURCE_DIRECTORY "${CTEST_DASHBOARD_ROOT}/CMake")
 SET (CTEST_BINARY_DIRECTORY "${CTEST_DASHBOARD_ROOT}/CMake-coverage")
 # Set the output file for bullsey
 SET (ENV{COVFILE} "${CTEST_BINARY_DIRECTORY}/CMake.cov")
 SET (CTEST_UPDATE_COMMAND    "cvs")
 SET (CTEST_CMAKE_GENERATOR "NMake Makefiles")
 SET (CTEST_PROJECT_NAME "CMake")
 
 # clear the binary directory and create an initial cache
 CTEST_EMPTY_BINARY_DIRECTORY (${CTEST_BINARY_DIRECTORY})
 # this is the initial cache to use for the binary tree, be careful to escape
 # any quotes inside of this string if you use it
 FILE(WRITE "${CTEST_BINARY_DIRECTORY}/CMakeCache.txt" "
 CTEST_TEST_CTEST:BOOL=1
 ")
 
 # Find cov01, the bullseye tool to turn on/off coverage
 FIND_PROGRAM(COV01 cov01)
 IF(NOT COV01)
  MESSAGE(FATAL_ERROR "Could not find cov01")
 ENDIF(NOT COV01)
 # turn off coverage for configure step
 SET(RES 1)
 EXECUTE_PROCESS(COMMAND ${COV01} -0 RESULT_VARIABLE RES)
 IF(RES)
  MESSAGE(FATAL_ERROR "could not run cov01 -0")
 ENDIF(RES)
 
 # do the dashboard steps
 CTEST_START (NightlyCoverage)
 CTEST_UPDATE (SOURCE "${CTEST_SOURCE_DIRECTORY}")
 CTEST_CONFIGURE (BUILD "${CTEST_BINARY_DIRECTORY}")
 # turn on coverage for build and test
 SET(RES 1)
 EXECUTE_PROCESS(COMMAND ${COV01} -1 RESULT_VARIABLE RES)
 IF(RES)
  MESSAGE(FATAL_ERROR "could not run cov01 -1")
 ENDIF(RES)
 CTEST_BUILD (BUILD "${CTEST_BINARY_DIRECTORY}")
 CTEST_TEST  (BUILD "${CTEST_BINARY_DIRECTORY}")
 CTEST_COVERAGE(BUILD "${CTEST_BINARY_DIRECTORY}")
 CTEST_SUBMIT ()
 EXECUTE_PROCESS(COMMAND ${COV01} -0 RESULT_VARIABLE RES)

Coverage With Python

Simple case

$ cat hello.py
"""
Comment #1
"""
def untested():
  print "untested"

"""
Comment #2
"""
def tested():
  print "tested"

"""
Comment #3
"""
if __name__ == '__main__':
  tested()

$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 hello.py

will produce a file

 hello.cover
 $ cat hello.cover
       """
       Comment #1
    1: """
    1: def untested():
         print "untested"
 
       """
       Comment #2
       """
    1: def tested():
    1:   print "tested"
 
       """
       Comment #3
       """
    1: if __name__ == '__main__':
    1:   tested()

Complex case

Using python code from VTK:

 VTK/Wrapping/Python/vtk/test/Testing.py

$ python /usr/lib/python2.3/trace.py -c --coverdir=. --ignore-dir /usr/lib/python2.3 Testing.py

will produce:

 ls *.cover
 Testing.cover       vtk.__init__.cover  vtk.filtering.cover         vtk.graphics.cover  vtk.imaging.cover  vtk.io.cover        vtk.rendering.cover      vtk.util.__init__.cover      vtk.util.vtkMethodParser.cover  vtk.widgets.cover
 vtk.__helper.cover  vtk.common.cover    vtk.genericfiltering.cover  vtk.hybrid.cover    vtk.infovis.cover  vtk.parallel.cover  vtk.test.BlackBox.cover  vtk.util.vtkConstants.cover  vtk.volumerendering.cover

Example:

Python Trace.py Coverage In CTest

The coverage was implemented, but there are two issues.

Minor issue #1: Trace.py does not differentiate between lines that are not covered and the lines that cannot possibly be covered. This means files with lots of comments will have poor coverage

Minor issue #2: Trace.py does not work with the compiled python files (*.pyc).

Major issue: Trace.py does not encode the location of the source file. This means that CTest has to guess where the source file is. Unfortunately the guessing can be either poor or time consuming.

Possible solution: Make wrapper for Trace.py and include it with CTest. This wrapper could solve the lack of source file location.



CMake: [Welcome | Site Map]