|
|
| (41 intermediate revisions by 14 users not shown) |
| Line 1: |
Line 1: |
| ==Introduction==
| | {{CMake/Template/Moved}} |
|
| |
|
| CTest is a testing tool distributed as a part of CMake. It can be used to
| | This page has moved [https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Testing-With-CTest here]. |
| automate updating (using CVS for example), configuring, building, testing,
| |
| performing memory checking, performing coverage, and submitting results to
| |
| [http://public.kitware.com/Dart Dart] dashboard system. This tutorial will
| |
| introduce the testing with CTest.
| |
| | |
| ==Simple Testing==
| |
| | |
| The easiest way to create CTest input files is using CMake. It has support for
| |
| adding tests in the project. To do that, insert the following command in the
| |
| CMakeLists.txt file:
| |
| | |
| ENABLE_TESTING()
| |
| | |
| From that point on, you can add tests in the project using ADD_TEST command:
| |
| | |
| ADD_TEST(SimpleTest ${EXECUTABLE_OUTPUT_PATH}/SimpleTest Hello)
| |
| | |
| After building the project, you should be able to run CTest on the project.
| |
| | |
| 1. When using Makefile generators, such as Unix Makefiles, Borland Makefiles,
| |
| or NMake Makefiles, the CTest can simply be run by running:
| |
| | |
| make test
| |
| | |
| 2. On GUI development environments such as Visual Studio rebuilding the target
| |
| RUN_TESTS will run tests on the project.
| |
| | |
| For more information about ENABLE_TESTING and ADD_TEST, Look at
| |
| [http://www.cmake.org/HTML/Documentation.html CMake Documentation] or run: | |
| | |
| cmake --help
| |
| cmake --help-full
| |
| cmake --help-command-list
| |
| | |
| or
| |
| | |
| cmake --help-command ENABLE_TESTING
| |
| cmake --help-command ADD_TEST
| |
| | |
| ==Dashboards==
| |
| | |
| Testing Dashboards are web pages that display overview of the project testing.
| |
| The testing clients update, configure, and build the project, as well as run
| |
| some number of tests. The results of these operations are then submitted to the
| |
| central server, which prepares the overview pages. Examples of Testing
| |
| Dashboards are:
| |
| [http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
| |
| VTK],
| |
| [http://www.itk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
| |
| ITK], and
| |
| [http://www.paraview.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
| |
| ParaView].
| |
| | |
| There are three types of dashboard submissions. Experimental submission is
| |
| submission of the current state of the project. It can be performed at any time
| |
| and will appear on the dashboard on the next roll-up. Nightly submission is
| |
| similar to experimental, except that the extra update will be performed using
| |
| the last nighty time. This way all nightly dashboard submissions correspond to
| |
| the state of the project at the same point in time. Finally Continuous
| |
| submissions are similar to Experimental submissions, except that an update is
| |
| performed and if any files were modified, the full dashboard testing is
| |
| performed.
| |
| | |
| ===Dashboard Preparation Using CMake===
| |
| | |
| To enable submission to Dart Testing Dashboard, include the following in
| |
| CMakeLists.txt:
| |
| | |
| INCLUDE(Dart)
| |
| | |
| By default the settings will be to submit to [http://www.kitware.com Kitware's]
| |
| [http://public.kitware.com/Public/Dashboard/MostRecentResults-Nightly/Dashboard.html
| |
| Public Dashboard]. In order to submit
| |
| to some other dashboard, create file "DartConfig.cmake" in the toplevel source directory and set the dashboard preferences.
| |
| | |
| Example of this file is:
| |
| | |
| <pre>
| |
| # Dashboard is opened for submissions for a 24 hour period starting at
| |
| # the specified NIGHLY_START_TIME. Time is specified in 24 hour format.
| |
| SET (NIGHTLY_START_TIME "23:00:00 EDT")
| |
| | |
| # Dart server to submit results (used by client)
| |
| IF(DROP_METHOD MATCHES http)
| |
| SET (DROP_SITE "public.kitware.com")
| |
| SET (DROP_LOCATION "/cgi-bin/HTTPUploadDartFile.cgi")
| |
| ELSE(DROP_METHOD MATCHES http)
| |
| SET (DROP_SITE "public.kitware.com")
| |
| SET (DROP_LOCATION "/incoming")
| |
| SET (DROP_SITE_USER "ftpuser")
| |
| SET (DROP_SITE_PASSWORD "public")
| |
| ENDIF(DROP_METHOD MATCHES http)
| |
| | |
| SET (TRIGGER_SITE
| |
| "http://${DROP_SITE}/cgi-bin/Submit-vtk-TestingResults.pl")
| |
| </pre>
| |
| | |
| This will submit testing results to the
| |
| [http://www.vtk.org/Testing/Dashboard/MostRecentResults-Nightly/Dashboard.html
| |
| VTK dashboard].
| |
| | |
| ===CTest - Client for Dart Dashboard Server===
| |
| | |
| CTest started as a simple tool to run some number of tests in the project, but
| |
| has evolved in the full ''Dart'' compatible client. It can perform simple task
| |
| of running a set of tests but it can also generate and submit Dart compatible
| |
| Dashboard results. The good thing about CTest is that it is self sustained. All
| |
| you need to do testing is CTest. Also, if you use CMake, you already have
| |
| CTest. Detailed description of CTest option can be seen by running:
| |
| | |
| ctest --help
| |
|
| |
| or
| |
| | |
| ctest --help-full
| |
| | |
| A simple way to submit Experimental dashboard is:
| |
| | |
| ctest -D Experimental
| |
| | |
| This will configure project, build project and check for any warnings or
| |
| errors, run tests if any tests are specified, run coverage if any coverage
| |
| files are specified, and submit to the specified ''Dart'' server.
| |
| | |
| To convert existing Dart Client run from the project, find lines like:
| |
| | |
| <pre>
| |
| cd ProjectNightlyBuildDirectory
| |
| tclsh /location/of/Dart/Source/Client/DashboardManager.tcl DartConfiguration.tcl \
| |
| Nightly Start Update Configure Build Test Submit
| |
| </pre>
| |
| | |
| and convert them to CTest style:
| |
| | |
| cd ProjectNightlyBuildDirectory
| |
| ctest -D Nightly
| |
| | |
| Dashboard can be also generated in stages. This way partial testing results can be submitted and seen before long operations are completed:
| |
| | |
| cd ProjectNightlyBuildDirectory
| |
| ctest -D NightlyStart
| |
| ctest -D NightlyUpdate
| |
| ctest -D NightlyConfigure
| |
| ctest -D NightlyBuild
| |
| ctest -D NightlySubmit
| |
| ctest -D NightlyTest
| |
| ctest -D NightlyCoverage
| |
| ctest -D NightlySubmit
| |
| ctest -D NightlyMemCheck
| |
| ctest -D NightlySubmit
| |
| | |
| Finally CTest extends functionality of Dart by adding:
| |
| | |
| # HTTP submission support
| |
| # Customization of the testing by providing:
| |
| #* Custom build error/warning regular expressions
| |
| #* Ability to suppress some tests from being tested or memory checked and ability to run subset of tests
| |
| #* Ability to run commands before and after tests are run
| |
| # Ability to run whole testing process described in a single script
| |
| | |
| For an example of how CTest can run the whole testing process described in a
| |
| single script, look at how CMake dashboards are created with the
| |
| [http://www.cmake.org/HTML/TestingSetup.html CTest -S script].
| |
| | |
| More information about CTest can be found in
| |
| [http://www.kitware.com/products/cmakebook.html Mastering CMake].
| |
| | |
| ==Conclusion==
| |
| | |
| Performing tests on the project is a great software development practice and can result in significant improvement on the quality of the project. CTest provides a simple and reliably way of performing nightly, continuous, and experimental tests.
| |
