|
|
| (10 intermediate revisions by 6 users not shown) |
| Line 1: |
Line 1: |
| ==Testing With CTest==
| | {{CMake/Template/Moved}} |
|
| |
|
| In [[CMake Testing With CTest|Testing With CTest]], you learned how to
| | This page has moved [https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Scripting-Of-CTest here]. |
| use CTest to perfom simple testing. That page also describes how to use
| |
| CTest to submit dashboards. The typical sequence of commands that
| |
| dashboard contributor will do involves:
| |
| | |
| cd /location/of/binary/Directory
| |
| rm -rf BinaryDirectory
| |
| mkdir BinaryDirectory
| |
| cd BinaryDirectory
| |
| ccmake /location/of/source/Directory
| |
| ctest -D Nightly
| |
| | |
| This is done on a typical UNIX system. On Windows system, the
| |
| contributor needs to do something similar using Windows command
| |
| interpreter, using something like Cygwin, or using a scripting language
| |
| such as Perl, Python, or Tcl. Unfortunately none of these solutions is
| |
| portable.
| |
| | |
| ==CTest Scripting==
| |
| | |
| CTest has a built-in scripting mode, which significantly simplifies
| |
| generating dashboards. Here is example of this script:
| |
| | |
| <pre>
| |
| SET (CTEST_SOURCE_DIRECTORY "C:/Hoffman/My Builds/CMake")
| |
| SET (CTEST_BINARY_DIRECTORY "C:/Hoffman/My Builds/CMakeVSNMake71")
| |
| | |
| SET (CTEST_CVS_COMMAND "C:/cygwin/bin/cvs.exe")
| |
| SET (CTEST_CVS_CHECKOUT "${CTEST_CVS_COMMAND} -d:pserver:hoffman@www.cmake.org:/cvsroot/CMake co -d\"${CTEST_SOURCE_DIRECTORY}\" CMake")
| |
| | |
| # which ctest command to use for running the dashboard
| |
| SET (CTEST_COMMAND
| |
| "C:/Program Files/CMake/bin/ctest.exe -D Nightly"
| |
| )
| |
| | |
| # what cmake command to use for configuring this dashboard
| |
| SET (CTEST_CMAKE_COMMAND
| |
| "C:/Program Files/CMake/bin/cmake.exe"
| |
| )
| |
| | |
| | |
| ####################################################################
| |
| # The values in this section are optional you can either
| |
| # have them or leave them commented out
| |
| ####################################################################
| |
| | |
| # should ctest wipe the binary tree before running
| |
| SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
| |
| | |
| # this is the initial cache to use for the binary tree, be careful to escape
| |
| # any quotes inside of this string if you use it
| |
| SET (CTEST_INITIAL_CACHE "
| |
| MAKECOMMAND:STRING=nmake -i
| |
| CMAKE_MAKE_PROGRAM:FILEPATH=nmake
| |
| CMAKE_GENERATOR:INTERNAL=NMake Makefiles
| |
| BUILDNAME:STRING=Win32-nmake71
| |
| SITE:STRING=VOGON.kitware
| |
| CVSCOMMAND:FILEPATH=C:/cygwin/bin/cvs.exe
| |
| ")
| |
| | |
| # set any extra environment variables to use during the execution of the script here:
| |
| SET (CTEST_ENVIRONMENT
| |
| )
| |
| </pre>
| |
| | |
| === with git ===
| |
| | |
| <pre>
| |
| set(CTEST_SOURCE_DIRECTORY "$ENV{HOME}/dashboards/libssh/source")
| |
| set(CTEST_BINARY_DIRECTORY "$ENV{HOME}/dashboards/libssh/build")
| |
| | |
| set(CTEST_SITE "magrathea.cynapses.org")
| |
| set(CTEST_BUILD_NAME "openssl-ssh1")
| |
| | |
| set(CTEST_CMAKE_GENERATOR "Unix Makefiles")
| |
| set(CTEST_BUILD_CONFIGURATION "Profiling")
| |
| set(CTEST_BUILD_OPTIONS "-DWITH_SSH1:BOOL=ON -DWITH_TESTING:BOOL=ON")
| |
| | |
| #######################################################################
| |
| | |
| ctest_empty_binary_directory(${CTEST_BINARY_DIRECTORY})
| |
| find_program(CTEST_GIT_COMMAND NAMES git)
| |
| | |
| if(NOT EXISTS "${CTEST_SOURCE_DIRECTORY}")
| |
| set(CTEST_CHECKOUT_COMMAND "${CTEST_GIT_COMMAND} clone git://git.libssh.org/projects/libssh/libssh.git ${CTEST_SOURCE_DIRECTORY}")
| |
| endif()
| |
| | |
| set(CTEST_UPDATE_COMMAND "${CTEST_GIT_COMMAND}")
| |
| set(CTEST_CONFIGURE_COMMAND "${CMAKE_COMMAND} -DCMAKE_BUILD_TYPE=${CTEST_BUILD_CONFIGURATION} ${CTEST_BUILD_OPTIONS}
| |
| \"-G${CTEST_CMAKE_GENERATOR}\" \"${CTEST_SOURCE_DIRECTORY}\"")
| |
| | |
| ctest_start("Nightly")
| |
| ctest_update()
| |
| ctest_configure()
| |
| ctest_build()
| |
| ctest_test()
| |
| #ctest_coverage()
| |
| #ctest_memcheck()
| |
| ctest_submit()
| |
| </pre>
| |
| | |
| To use SVN, just change the ''CVSCOMMAND'' line by ''SVNCOMMAND'' and give the right path to the ''svn.exe'' program. Also, don't forget to change the ''LC_MESSAGES'' environment variable to ''en_EN'' in your script or directly in your system. This will force the SVN output language to english so that CTest will be able to analyse the data from SVN and submit them correctly to Dart2.
| |
| | |
| <pre>
| |
| SET( $ENV{LC_MESSAGES} "en_EN" )
| |
| </pre>
| |
| | |
| This script defined all the information CTest needs to submit CMake
| |
| dashboard on Windows XP computer using NMake build system. The syntax
| |
| used is the same as in CMake. Also, most commands available in CMake are
| |
| available here, except things that require generator, such as
| |
| TRY_COMPILE, ADD_LIBRARY, etc.
| |
| | |
| The script can be run by executing:
| |
| | |
| ctest -S /path/to/script.cmake
| |
| | |
| or in this case:
| |
| | |
| c:/Program Files/CMake/bin/ctest -S c:/Hoffman/DashboardScripts/vogon_cmake_nmake.cmake
| |
| | |
| ===Main Settings===
| |
| | |
| Every CTest script has to contain the source and binary directory:
| |
| | |
| SET (CTEST_SOURCE_DIRECTORY "$ENV{HOME}/Dashboards/My Testing/Insight")
| |
| SET (CTEST_BINARY_DIRECTORY "$ENV{HOME}/Dashboards/My Testing/Insight-bin")
| |
| | |
| The "$ENV{HOME}" gets replaced by the environment variable "HOME", which
| |
| on most systems points to user's home directory.
| |
| | |
| Second thing we need is the command that perform initial configuration
| |
| of the project. For example, if the project uses CMake, then the command
| |
| would be something like:
| |
| | |
| SET (CTEST_CMAKE_COMMAND "/usr/local/bin/cmake")
| |
| | |
| That assumes that CMake is installed in /usr/local. Similarly to setting
| |
| up CMake command, we need to setup the way CTest will be called. Let say
| |
| in your case you run CTest using:
| |
| | |
| ctest -D Nightly
| |
| | |
| Then the command in the CTest script should be:
| |
| | |
| SET (CTEST_COMMAND "/usr/local/bin/ctest -D Nightly")
| |
| | |
| The final important piece we need is the initial cache that CMake uses
| |
| to configure the project. This should be minimal set of settings that
| |
| will allow the system to test certain aspect of the project. Example
| |
| initial cache would be:
| |
| | |
| <pre>
| |
| SET (CTEST_INITIAL_CACHE "
| |
| //Name of generator.
| |
| CMAKE_GENERATOR:INTERNAL=Visual Studio 7
| |
| | |
| //Name of the build
| |
| BUILDNAME:STRING=Win32-vs70
| |
| | |
| //Name of the computer/site where compile is being run
| |
| SITE:STRING=DASH2.kitware
| |
| | |
| //Build VTK with shared libraries.
| |
| BUILD_SHARED_LIBS:BOOL=ON
| |
| | |
| //Path to the CVS
| |
| CVSCOMMAND:FILEPATH=C:/cygwin/bin/cvs.exe
| |
| | |
| //ITK TCL and Python wrapping
| |
| ITK_CSWIG_TCL:BOOL=ON
| |
| ITK_CSWIG_PYTHON:BOOL=ON
| |
| ")
| |
| </pre>
| |
| | |
| This will set the generator to be "Visual Studio 7", it will use shared
| |
| libraries and turn on TCL and Python wrapping. Also, in the cache we can
| |
| set the SITE and BUILDNAME variables, which will be used to display the
| |
| testing results on the dashboard.
| |
| | |
| If you do not use default generator (such as Unix Makefiles), make sure you specify one. On systems that have multimple generators, it is a good idea to specify one always, since otherwise CMake might pick the wrong one.
| |
| | |
| Available generators are:
| |
| | |
| {| border="1" cellpadding="2" cellspacing="0"
| |
| |- bgcolor="#abcdef"
| |
| ! Generator !! Description
| |
| |-
| |
| | Unix Makefiles
| |
| | Unix style makefiles that can be processed using most Make processors except nonconforming processors such as NMake and Borland Make.
| |
| |-
| |
| | Borland Makefiles
| |
| | Makefiles that can be processed using Borland Make processor
| |
| |-
| |
| | NMake Makefiles
| |
| | Makefiles that can be processed using NMake Make processor
| |
| |-
| |
| | Visual Studio 6
| |
| | Visual studio 6 project files
| |
| |-
| |
| | Visual Studio 7
| |
| | Visual Studio 7 project files (not same as Visual Studio 7 .NET 2003
| |
| |-
| |
| | Visual Studio 7 .NET 2003
| |
| | Visual Studio 7 .NET 2003 project files
| |
| |-
| |
| | Visual Studio 8 2005
| |
| | Visual Studi 8 (aka Visual Studio 2005) project files.
| |
| |}
| |
| | |
| | |
| This is it. The CTest script can now be tested using:
| |
| | |
| ctest -S /path/to/script/script.cmake -V
| |
| | |
| The -V argument tells CTest to be extra verbose when doing dashboard.
| |
| | |
| ===More Settings===
| |
| | |
| The main settings will generate dashboard for most systems. There are
| |
| however issues in certain setups.
| |
| | |
| Most dashboard contributors want to start dashboard and not touch those
| |
| systems for months. The problem is that the binary directory of the
| |
| project will grow larger and larger. Common practice therefore is to
| |
| wipe out binary directory every time dashboard is run. We can achieve
| |
| this in CTest script by setting:
| |
| CTEST_START_WITH_EMPTY_BINARY_DIRECTORY:
| |
| | |
| SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
| |
| | |
| Now CTest will wipe out the binary tree every time. Make sure that the
| |
| source tree is not equal to binary tree.
| |
| | |
| Second useful option is setting of environment in which CTest runs. Let
| |
| say you have Cygwin, Borland, Visual Studio, and Intel compiler on your
| |
| Windows system and you do not want to have system's global PATH pointing
| |
| to various directories (Borland and Cygwin use "make", which one should
| |
| be used). Now you want to do Borland dashboard. You can set environment
| |
| variables using CTEST_ENVIRONMENT:
| |
| | |
| SET (CTEST_ENVIRONMENT
| |
| "PATH=c:/WINDOWS/system32\;c:/WINDOWS\;c:/Programs/Borland/Bcc55/Bin")
| |
| | |
| Note that CTEST_ENVIRONMENT can accept multiple environment variables.
| |
| For example:
| |
| | |
| SET (CTEST_ENVIRONMENT
| |
| "DISPLAY=:0"
| |
| "GLIBCPP_FORCE_NEW=1"
| |
| "CC=gcc-3.4"
| |
| "CXX=g++-3.4"
| |
| )
| |
| | |
| Another useful thing to do is to perform extra update on external
| |
| repository. For example, VTK dashboard need to update VTK data. This can
| |
| be achieved using CTEST_EXTRA_UPDATES_*. This also requires the
| |
| CTEST_CVS_COMMAND to be set. An example of extra update is:
| |
| | |
| SET (CTEST_CVS_COMMAND "/usr/bin/cvs")
| |
| SET (CTEST_EXTRA_UPDATES_1 "$ENV{HOME}/Dashboards/MyTests/VTKData" "-dAP")
| |
| | |
| There can be up to ten CTEST_EXTRA_UPDATES_* settings:
| |
| CTEST_EXTRA_UPDATES_1, CTEST_EXTRA_UPDATES_2, ...,
| |
| CTEST_EXTRA_UPDATES_10.
| |
| | |
| Extra update can also be used to update DartConfig.cmake to the latest version, to receive settings right away instead of waiting until next day. An example of those kind of settings is a nightly start time.
| |
| | |
| SET (CTEST_EXTRA_UPDATES_2 "${CTEST_SOURCE_DIRECTORY}" " -dAP DartConfig.cmake")
| |
| | |
| ===Advanced Settings===
| |
| | |
| When performing complex dashboards setups, some more CMake/CTest
| |
| knowledge is needed. This section will describe solutions to some common
| |
| CTest problems.
| |
| | |
| ====Memory Checking====
| |
| | |
| Doing memory checking using Valgrind or similar product can take
| |
| significant amount of time. There are two improvements the dashboard
| |
| contributor can do to make dashboard more useful. First one is to submit
| |
| partial results. This can be achieved by breaking down the CTest run
| |
| into individual runs:
| |
| | |
| SET (CTEST_COMMAND
| |
| "/usr/local/bin/ctest -D NightlyStart -D NightlyUpdate -D NightlyConfigure -D NightlyBuild -D NightlyTest -D NightlySubmit"
| |
| "/usr/local/bin/ctest -D NightlyMemCheck -D NightlySubmit"
| |
| )
| |
| | |
| The first CTest command will perform standard Nightly dashboard.
| |
| The last two will perform only Memory checking and submission. This the
| |
| whole dashboard submission will be there except the memory checking.
| |
| Once the memory checking is completed, it will be submitted.
| |
| | |
| The second thing is to add special flags to make the CTest only test
| |
| subset of tests:
| |
| | |
| SET (CTEST_COMMAND
| |
| ...
| |
| "/usr/local/bin/ctest -D NightlyMemCheck -I 0,,17"
| |
| "/usr/local/bin/ctest -D NightlySubmit"
| |
| )
| |
| | |
| The "-I 0,,17" will tell CTest to only run memory checking of every 17th
| |
| test.
| |
| | |
| ====Initial Checkout====
| |
| | |
| Let say you do not want to require the source of the project to be
| |
| available. This is useful if the dashboards are performed on the
| |
| temporary drive that can get erased. The following setting will checkout
| |
| the source tree of CMake out of CVS repository before the testing is
| |
| performed:
| |
| | |
| SET (CTEST_CVS_CHECKOUT
| |
| "${CTEST_CVS_COMMAND} -q -z3
| |
| -d :pserver:anoncvs@www.cmake.org:/cvsroot/CMake
| |
| co -d \"CMake\" -D yesterday CMake")
| |
| | |
| Another useful feature is to backup source tree before performing
| |
| dashboard. This way if the build fails, you can restore the old tree.
| |
| This is useful when you require the source tree to be in good condition.
| |
| This option has to be used in combination with CTEST_CVS_CHECKOUT. To
| |
| turn this feature on, add this to CTest script:
| |
| | |
| SET (CTEST_BACKUP_AND_RESTORE TRUE)
| |
| | |
| ====Continuous Builds (old Style) ====
| |
| | |
| When setting up Continuous dashboard, the CTest has to be run
| |
| periodically to check if there were any changes in the source. When the
| |
| change appears, the full dashboard is tested. This can be achieved by
| |
| running CTest from system scheduler or some script, or by using
| |
| Continuous feature of CTest script. First of all, make sure that
| |
| CTEST_COMMAND is using Continuous model:
| |
| | |
| SET (CTEST_COMMAND
| |
| "C:/Program Files/CMake20/bin/ctest.exe -D Continuous"
| |
| )
| |
| | |
| Also, if the number of tests is high, you may want to only run subset of
| |
| tests:
| |
| | |
| SET (CTEST_COMMAND
| |
| "C:/Program Files/CMake20/bin/ctest.exe -D Continuous -I ,,3"
| |
| )
| |
| | |
| Now set the continuous parameters:
| |
| | |
| SET (CTEST_CONTINUOUS_DURATION 600)
| |
| SET (CTEST_CONTINUOUS_MINIMUM_INTERVAL 10)
| |
| SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY_ONCE 1)
| |
| | |
| The CTEST_CONTINUOUS_DURATION is the duration of Continuous testing in
| |
| minutes. In this case it will run Continuous dashboards for 10 hours. If
| |
| it starts at 8 AM, the last continuous will be done around 6 PM. The
| |
| CTEST_CONTINUOUS_MINIMUM_INTERVAL is the minimum interval between
| |
| Continuous checks. In this case it will perform check every 10 minutes.
| |
| CTEST_START_WITH_EMPTY_BINARY_DIRECTORY_ONCE will remove binary tree
| |
| once the first Continuous is run. This way Continuous tests will take
| |
| less time, since it will only rebuild modified files.
| |
| | |
| ====Continuous Builds (new Style) ====
| |
| | |
| CTEST_CONTINUOUS_DURATION is not used in the new style scripts. Instead, you you should use something like this:
| |
| | |
| while (${CTEST_ELAPSED_TIME} LESS 36000)
| |
| set (START_TIME ${CTEST_ELAPSED_TIME})
| |
| ctest_start (Continuous)
| |
| ....
| |
| ctest_sleep( ${START_TIME} 300 ${CTEST_ELAPSED_TIME})
| |
| endwhile()
| |
| | |
| ===Available Variables===
| |
| | |
| When running CTest scripts, there are several variables available that
| |
| can simplify writing CTest scripts. These variable include:
| |
| | |
| {| border="1" cellpadding="2" cellspacing="0"
| |
| |- bgcolor="#abcdef"
| |
| ! Variable !! Description
| |
| |-
| |
| | CTEST_SCRIPT_DIRECTORY
| |
| | Location of directory in which the CTest script is
| |
| |-
| |
| | CTEST_SCRIPT_NAME
| |
| | Name of the CTest script
| |
| |- valign="top"
| |
| | CTEST_SCRIPT_ARG
| |
| | The extra arguments passed to CTest -S. For example, when running:
| |
| | |
| <pre>
| |
| ctest -S /home/scripts/mysystem.cmake,Experimental
| |
| </pre>
| |
| | |
| then the CTEST_SCRIPT_ARG will be set to Experimental. Then this can be
| |
| used inside the script:
| |
| | |
| <pre>
| |
| SET(MODEL Nightly)
| |
| IF(${CTEST_SCRIPT_ARG} MATCHES Experimental)
| |
| SET(MODEL Experimental)
| |
| ENDIF(${CTEST_SCRIPT_ARG} MATCHES Experimental)
| |
| SET (CTEST_COMMAND
| |
| "/usr/local/bin/ctest -D ${MODEL}")
| |
| </pre>
| |
| | |
| Now we can specify Nighlty or Experimental without editing CTest script.
| |
| |-
| |
| |}
| |
| | |
| ==Setting Up Cron/Scheduler==
| |
| | |
| ===On Unix/MacOSX===
| |
| | |
| Most Unix and Unix like systems use '''cron'''. Cron is a daemon that runs programs at specified times as described in crontab. Example of crontab is:
| |
| | |
| 1 22 * * * /usr/bin/ctest -S /dash/dash1.cmake -V > /dash/logs/tests.log 2>&1
| |
| | |
| The part of the line:
| |
|
| |
| 1 22 * * *
| |
| | |
| specifies when to run the scheduled task. The columns correspond to minutes, hours, days, months, day of the week. This gets translated to 10:01 PM every day, every month.
| |
| Another example would be:
| |
| | |
| 47 6 * * 7
| |
| | |
| which is 6:47 AM on sunday.
| |
| | |
| The end of the line:
| |
| | |
| > /dash/logs/tests.log 2>&1
| |
| | |
| is korn-shell (/bin/sh) syntax for sending all messages (standard output and standard error) to a file '''/dash/logs/tests.log'''. If you set different shell in your crontab, make sure to use apropriate syntax.
| |
| | |
| To enable crontab, use the following command:
| |
| | |
| crontab -e
| |
| | |
| ===On Windows / Cygwin / MinGW===
| |
| | |
| The following images are generated on Windows XP Pro, but the concepts should be the same on all Windows systems. Make sure to enable password for the user, otherwise scheduled tasks will not work.
| |
| | |
| 1. Open "Scheduled Tasks" from Control Panel.
| |
| | |
| [[Image:Sc_task_scheduler.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 2. Select '''Add Scheduled Task'''
| |
| | |
| [[Image:Sc_start_new_scheduled_task.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 3. Select '''Next''' to select command.
| |
| | |
| [[Image:Sc_select_browse.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 4. Click '''Browse...''' and select '''CTest.exe'''.
| |
| | |
| [[Image:Sc_specify_command.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 5. Click '''Next''' and select name and repetition date. Repetition date for Nightly dashboards should be '''Daily'''.
| |
| | |
| [[Image:Sc_specified_name_and_repetition.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 6. Click '''Next''' and select time to start the dashboard.
| |
| | |
| [[Image:Sc_set_time.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 7. Click '''Next''' and select '''Open advanced properties...''' to fine tune the scheduled task.
| |
| | |
| [[Image:Sc_open_advanced_settings.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 8. Select '''Next''' and type password of the user.
| |
| | |
| [[Image:Sc_type_password.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| | |
| 9. Task is created. The ''Advanced Properties'' dialog should open.
| |
| | |
| [[Image:Sc_task_created.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 10. In advanced properties, specify full command name. This is very important that you use double quotes in case you have space in your path.
| |
| | |
| "c:/Path to ctest/ctest.exe" -S c:/Path to Dashboard Scripts/dashboard_script.cmake -V
| |
| | |
| [[Image:Sc_proper_command_line.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| | |
| 11. Select '''Ok'', which will ask for password again.
| |
| | |
| [[Image:Sc_finish_password.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| 12. The new task should be created.
| |
| | |
| [[Image:Sc_new_task_exists.png|left|Task Scheduler]]
| |
| <br clear="all">
| |
| | |
| ==Conclusion==
| |
| | |
| CTest scripts can significantly simplify dashboard contribution. Using
| |
| simple template script with slight modification, the whole cluster of
| |
| various systems can submit dashboards.
| |
| | |
| {{CMake/Template/Footer}}
| |
