|
|
| (40 intermediate revisions by 14 users not shown) |
| Line 1: |
Line 1: |
| ==Testing With CTest==
| | {{CMake/Template/Moved}} |
|
| |
|
| In [[CMake Testing With CTest|Testing With CTest]], you learned how to
| | This page has moved [https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Scripting-Of-CTest here]. |
| use CTest to perfom simple testing. That page also describes how to use
| |
| CTest to submit dashboards. The typical sequence of commands that
| |
| dashboard contributor will do involves:
| |
| | |
| cd /location/of/binary/Directory
| |
| rm -rf BinaryDirectory
| |
| mkdir BinaryDirectory
| |
| cd BinaryDirectory
| |
| ccmake /location/of/source/Directory
| |
| ctest -D Nightly
| |
| | |
| This is done on a typical UNIX system. On Windows system, the
| |
| contributor needs to do something similar using Windows command
| |
| interpreter, using something like Cygwin, or using a scripting language
| |
| such as Perl, Python, or Tcl. Unfortunately none of these solutions is
| |
| portable.
| |
| | |
| ==CTest Scripting==
| |
| | |
| CTest has a built-in scripting mode, which significantly simplifies
| |
| generating dashboards. Here is example of this script:
| |
| | |
| <pre>
| |
| SET (CTEST_SOURCE_DIRECTORY "C:/Hoffman/My Builds/CMake")
| |
| SET (CTEST_BINARY_DIRECTORY "C:/Hoffman/My Builds/CMakeVSNMake71")
| |
| | |
| SET (CTEST_CVS_COMMAND "C:/cygwin/bin/cvs.exe")
| |
| SET (CTEST_CVS_CHECKOUT "${CTEST_CVS_COMMAND} -d
| |
| :pserver:hoffman@www.cmake.org:/cvsroot/CMake co -d
| |
| \"${CTEST_SOURCE_NAME}\" CMake")
| |
| | |
| # which ctest command to use for running the dashboard
| |
| SET (CTEST_COMMAND
| |
| "C:/Program Files/CMake/bin/ctest.exe -D Nightly"
| |
| )
| |
| | |
| # what cmake command to use for configuring this dashboard
| |
| SET (CTEST_CMAKE_COMMAND
| |
| "C:/Program Files/CMake/bin/cmake.exe"
| |
| )
| |
| | |
| | |
| ####################################################################
| |
| # The values in this section are optional you can either
| |
| # have them or leave them commented out
| |
| ####################################################################
| |
| | |
| # should ctest wipe the binary tree before running
| |
| SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
| |
| | |
| # this is the initial cache to use for the binary tree, be careful to
| |
| escape
| |
| # any quotes inside of this string if you use it
| |
| SET (CTEST_INITIAL_CACHE "
| |
| MAKECOMMAND:STRING=nmake -i
| |
| CMAKE_MAKE_PROGRAM:FILEPATH=nmake
| |
| CMAKE_GENERATOR:INTERNAL=NMake Makefiles
| |
| BUILDNAME:STRING=Win32-nmake71
| |
| SITE:STRING=VOGON.kitware
| |
| CVSCOMMAND:FILEPATH=C:/cygwin/bin/cvs.exe
| |
| ")
| |
| | |
| # set any extra environment variables here
| |
| SET (CTEST_ENVIRONMENT
| |
| )
| |
| </pre>
| |
| | |
| This script defined all the information CTest needs to submit CMake
| |
| dashboard on Windows XP computer using NMake build system. The syntax
| |
| used is the same as in CMake. Also, most commands available in CMake are
| |
| available here, except things that require generator, such as
| |
| TRY_COMPILE, ADD_LIBRARY, etc.
| |
| | |
| The script can be run by executing:
| |
| | |
| ctest -S /path/to/script.cmake
| |
| | |
| or in this case:
| |
| | |
| c:/Program Files/CMake/bin/ctest -S
| |
| c:/Hoffman/DashboardScripts/vogon_cmake_nmake.cmake
| |
| | |
| ===Main Settings===
| |
| | |
| Every CTest script has to contain the source and binary directory:
| |
| | |
| SET (CTEST_SOURCE_DIRECTORY "$ENV{HOME}/Dashboards/My
| |
| Testing/Insight")
| |
| SET (CTEST_BINARY_DIRECTORY "$ENV{HOME}/Dashboards/My
| |
| Testing/Insight-bin")
| |
| | |
| The "$ENV{HOME}" gets replaced by the environment variable "HOME", which
| |
| on most systems points to user's home directory.
| |
| | |
| Second thing we need is the command that perform initial configuration
| |
| of the project. For example, if the project uses CMake, then the command
| |
| would be something like:
| |
| | |
| SET (CTEST_CMAKE_COMMAND "/usr/local/bin/cmake")
| |
| | |
| That assumes that CMake is installed in /usr/local. Similarly to setting
| |
| up CMake command, we need to setup the way CTest will be called. Let say
| |
| in your case you run CTest using:
| |
| | |
| ctest -D Nightly
| |
| | |
| Then the command in the CTest script should be:
| |
| | |
| SET (CTEST_COMMAND "/usr/local/bin/ctest -D Nightly")
| |
| | |
| The final important piece we need is the initial cache that CMake uses
| |
| to configure the project. This should be minimal set of settings that
| |
| will allow the system to test certain aspect of the project. Example
| |
| initial cache would be:
| |
| | |
| <pre>
| |
| SET (CTEST_INITIAL_CACHE "
| |
| //Name of generator.
| |
| CMAKE_GENERATOR:INTERNAL=Visual Studio 7
| |
| | |
| //Name of the build
| |
| BUILDNAME:STRING=Win32-vs70
| |
| | |
| //Name of the computer/site where compile is being run
| |
| SITE:STRING=DASH2.kitware
| |
| | |
| //Build VTK with shared libraries.
| |
| BUILD_SHARED_LIBS:BOOL=ON
| |
| | |
| //Path to the CVS
| |
| CVSCOMMAND:FILEPATH=C:/cygwin/bin/cvs.exe
| |
| | |
| //ITK TCL and Python wrapping
| |
| ITK_CSWIG_TCL:BOOL=ON
| |
| ITK_CSWIG_PYTHON:BOOL=ON
| |
| ")
| |
| </pre>
| |
| | |
| This will set the generator to be "Visual Studio 7", it will use shared
| |
| libraries and turn on TCL and Python wrapping. Also, in the cache we can
| |
| set the SITE and BUILDNAME variables, which will be used to display the
| |
| testing results on the dashboard.
| |
| | |
| This is it. The CTest script can now be tested using:
| |
| | |
| ctest -S /path/to/script/script.cmake -V
| |
| | |
| The -V argument tells CTest to be extra verbose when doing dashboard.
| |
| | |
| ===More Settings===
| |
| | |
| The main settings will generate dashboard for most systems. There are
| |
| however issues in certain setups.
| |
| | |
| Most dashboard contributors want to start dashboard and not touch those
| |
| systems for months. The problem is that the binary directory of the
| |
| project will grow larger and larger. Common practice therefore is to
| |
| wipe out binary directory every time dashboard is run. We can achieve
| |
| this in CTest script by setting:
| |
| CTEST_START_WITH_EMPTY_BINARY_DIRECTORY:
| |
| | |
| SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
| |
| | |
| Now CTest will wipe out the binary tree every time. Make sure that the
| |
| source tree is not equal to binary tree.
| |
| | |
| Second useful option is setting of environment in which CTest runs. Let
| |
| say you have Cygwin, Borland, Visual Studio, and Intel compiler on your
| |
| Windows system and you do not want to have system's global PATH pointing
| |
| to various directories (Borland and Cygwin use "make", which one should
| |
| be used). Now you want to do Borland dashboard. You can set environment
| |
| variables using CTEST_ENVIRONMENT:
| |
| | |
| SET (CTEST_ENVIRONMENT
| |
|
| |
| "PATH=c:/WINDOWS/system32\;c:/WINDOWS\;c:/Programs/Borland/Bcc55/Bin"
| |
| | |
| Note that CTEST_ENVIRONMENT can accept multiple environment variables.
| |
| For example:
| |
| | |
| SET (CTEST_ENVIRONMENT
| |
| "DISPLAY=:0"
| |
| "GLIBCPP_FORCE_NEW=1"
| |
| "CC=gcc-3.4"
| |
| "CXX=g++-3.4"
| |
| )
| |
| | |
| Another useful thing to do is to perform extra update on external
| |
| repository. For example, VTK dashboard need to update VTK data. This can
| |
| be achieved using CTEST_EXTRA_UPDATES_*. This also requires the
| |
| CTEST_CVS_COMMAND to be set. An example of extra update is:
| |
| | |
| SET (CTEST_CVS_COMMAND "/usr/bin/cvs")
| |
| SET (CTEST_EXTRA_UPDATES_1 "$ENV{HOME}/Dashboards/MyTests/VTKData"
| |
| "-dAP")
| |
| | |
| There can be up to ten CTEST_EXTRA_UPDATES_* settings:
| |
| CTEST_EXTRA_UPDATES_1, CTEST_EXTRA_UPDATES_2, ...,
| |
| CTEST_EXTRA_UPDATES_10.
| |
| | |
| ===Advanced Settings===
| |
| | |
| When performing complex dashboards setups, some more CMake/CTest
| |
| knowledge is needed. This section will describe solutions to some common
| |
| CTest problems.
| |
| | |
| Doing memory checking using Valgrind or similar product can take
| |
| significant amount of time. There are two improvements the dashboard
| |
| contributor can do to make dashboard more useful. First one is to submit
| |
| partial results. This can be achieved by breaking down the CTest run
| |
| into individual runs:
| |
| | |
| SET (CTEST_COMMAND
| |
| "/usr/local/bin/ctest -D NightlyStart"
| |
| "/usr/local/bin/ctest -D NightlyUpdate"
| |
| "/usr/local/bin/ctest -D NightlyConfigure"
| |
| "/usr/local/bin/ctest -D NightlyBuild"
| |
| "/usr/local/bin/ctest -D NightlyTest"
| |
| "/usr/local/bin/ctest -D NightlySubmit"
| |
| "/usr/local/bin/ctest -D NightlyMemCheck"
| |
| "/usr/local/bin/ctest -D NightlySubmit"
| |
| )
| |
| | |
| The first six CTest commands will perform standard Nightly dashboard.
| |
| The last two will perform only Memory checking and submission. This the
| |
| whole dashboard submission will be there except the memory checking.
| |
| Once the memory checking is completed, it will be submitted.
| |
| | |
| The second thing is to add special flags to make the CTest only test
| |
| subset of tests:
| |
| | |
| SET (CTEST_COMMAND
| |
| ...
| |
| "/usr/local/bin/ctest -D NightlyMemCheck -I 0,,17"
| |
| "/usr/local/bin/ctest -D NightlySubmit"
| |
| )
| |
| | |
| The "-I 0,,17" will tell CTest to only run memory checking of every 17th
| |
| test.
| |
| | |
| Let say you do not want to require the source of the project to be
| |
| available. This is useful if the dashboards are performed on the
| |
| temporary drive that can get erased. The following setting will checkout
| |
| the source tree of CMake out of CVS repository before the testing is
| |
| performed:
| |
| | |
| SET (CTEST_CVS_CHECKOUT
| |
| "${CTEST_CVS_COMMAND} -q -z3
| |
| -d :pserver:anoncvs@www.cmake.org:/cvsroot/CMake
| |
| co -d \"CMake\" -D yesterday CMake")
| |
| | |
| Another useful feature is to backup source tree before performing
| |
| dashboard. This way if the build fails, you can restore the old tree.
| |
| This is useful when you require the source tree to be in good condition.
| |
| This option has to be used in combination with CTEST_CVS_CHECKOUT. To
| |
| turn this feature on, add this to CTest script:
| |
| | |
| SET (CTEST_BACKUP_AND_RESTORE TRUE)
| |
| | |
| When setting up Continuous dashboard, the CTest has to be run
| |
| periodically to check if there were any changes in the source. When the
| |
| change appears, the full dashboard is tested. This can be achieved by
| |
| running CTest from system scheduler or some script, or by using
| |
| Continuous feature of CTest script. First of all, make sure that
| |
| CTEST_COMMAND is using Continuous model:
| |
| | |
| SET (CTEST_COMMAND
| |
| "C:/Program Files/CMake20/bin/ctest.exe -D Continuous"
| |
| )
| |
| | |
| Also, if the number of tests is high, you may want to only run subset of
| |
| tests:
| |
| | |
| SET (CTEST_COMMAND
| |
| "C:/Program Files/CMake20/bin/ctest.exe -D Continuous -I ,,3"
| |
| )
| |
| | |
| Now set the continuous parameters:
| |
| | |
| SET (CTEST_CONTINUOUS_DURATION 600)
| |
| SET (CTEST_CONTINUOUS_MINIMUM_INTERVAL 10)
| |
| SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY_ONCE 1)
| |
| | |
| The CTEST_CONTINUOUS_DURATION is the duration of Continuous testing in
| |
| minutes. In this case it will run Continuous dashboards for 10 hours. If
| |
| it starts at 8 AM, the last continuous will be done around 6 PM. The
| |
| CTEST_CONTINUOUS_MINIMUM_INTERVAL is the minimum interval between
| |
| Continuous checks. In this case it will perform check every 10 minutes.
| |
| CTEST_START_WITH_EMPTY_BINARY_DIRECTORY_ONCE will remove binary tree
| |
| once the first Continuous is run. This way Continuous tests will take
| |
| less time, since it will only rebuild modified files.
| |
| | |
| ===Available Variables===
| |
| | |
| When running CTest scripts, there are several variables available that
| |
| can simplify writing CTest scripts. These variable include:
| |
| | |
| {| border="1" cellpadding="2" cellspacing="0"
| |
| |- bgcolor="#abcdef"
| |
| ! Variable !! Description
| |
| |-
| |
| | CTEST_SCRIPT_DIRECTORY
| |
| | Location of directory in which the CTest script is
| |
| |-
| |
| | CTEST_SCRIPT_NAME
| |
| | Name of the CTest script
| |
| |- valign="top"
| |
| | CTEST_SCRIPT_ARG
| |
| | The extra arguments passed to CTest -S. For example, when running:
| |
| | |
| <pre>
| |
| ctest -S /home/scripts/mysystem.cmake,Experimental
| |
| </pre>
| |
| | |
| then the CTEST_SCRIPT_ARG will be set to Experimental. Then this can be
| |
| used inside the script:
| |
| | |
| <pre>
| |
| SET(MODEL Nightly)
| |
| IF(${CTEST_SCRIPT_ARGS} MATCHES Experimental)
| |
| SET(MODEL Experimental)
| |
| ENDIF(${CTEST_SCRIPT_ARGS} MATCHES Experimental)
| |
| SET (CTEST_COMMAND
| |
| "/usr/local/bin/ctest -D ${MODEL}")
| |
| </pre>
| |
| | |
| Now we can specify Nighlty or Experimental without editing CTest script.
| |
| |-
| |
| |}
| |
| | |
| | |
| ==Conclusion==
| |
| | |
| CTest scripts can significantly simplify dashboard contribution. Using
| |
| simple template script with slight modification, the whole cluster of
| |
| various systems can submit dashboards.
| |
