|
|
| (6 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| ==Introduction==
| | {{CMake/Template/Moved}} |
|
| |
|
| CTest is a powerful testing client compatible with
| | This page has moved [https://gitlab.kitware.com/cmake/community/wikis/doc/ctest/Generating-Testing-Files here]. |
| [http://public.kitware.com/Dart Dart] Dashboard system. It is distributed as a | |
| part of [http://www.cmake.org CMake] but can be used with or without CMake.
| |
| This tutorial explains some background behind the testing with CTest and the
| |
| process of generating CTest testing files.
| |
| | |
| ==CTest Overview==
| |
| | |
| CTest currently uses the old Dart input configuration file. New input format is
| |
| being developed, but the old format will be valid in the future. For legacy
| |
| purposes, the input files are called ''DartConfiguration.tcl'', and
| |
| ''DartTestfile.txt''. ''DartConfiguration.tcl'' describes the system on which
| |
| tests are performed, while ''DartTestfile.txt'' provides the list of tests.
| |
| | |
| CTest reads ''DartConfiguration.tcl'' and depending on the options provided it
| |
| performs certain stage of testing.
| |
| | |
| ===Description Of The System===
| |
| | |
| Even though the extension of ''DartConfiguration.tcl''
| |
| file is ''.tcl'', it is not TCL file. It is simply list of key/value pairs.
| |
| Line starting with ''#'' are considered comments and are ignored. This file has to appear at the toplevel directory of the project's build directory.
| |
| | |
| Example ''DartConfiguration.tcl'' file is:
| |
| | |
| <pre>
| |
| SourceDirectory: /path/to/project/source/directory
| |
| BuildDirectory: /path/to/project/build/directory
| |
| CVSUpdateOptions: -d -A -P
| |
| Site: my-system-name
| |
| BuildName: build-name
| |
| DropSite: public.kitware.com
| |
| DropLocation: /cgi-bin/HTTPUploadDartFile.cgi
| |
| DropMethod: http
| |
| TriggerSite: http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.pl
| |
| NightlyStartTime: 21:00:00 EDT
| |
| ConfigureCommand: "/path/to/source/directory/Project/configure"
| |
| MakeCommand: /usr/bin/make -i
| |
| CVSCommand: /usr/bin/cvs
| |
| CoverageCommand: /usr/bin/gcov
| |
| TimeOut: 1500
| |
| </pre>
| |
| | |
| ====DartConfiguration.tcl options====
| |
| {| border="1"
| |
| |- bgcolor="#abcdef"
| |
| ! Key !! Description !! Example value
| |
| |-
| |
| | SourceDirectory || Path to the source directory of the project. Used for CVS or SVN update. || /path/to/project/source/directory
| |
| |-
| |
| | BuildDirectory || Path to the build directory of the project. This is where Testing results will go and all testing commands will be run in this directory. || /path/to/project/build/directory
| |
| |-
| |
| | Site || Refers to the system name used to distinguish the dashboard entry. || my-system-name
| |
| |-
| |
| | BuildName || A breif description of the system and the project. || FLTK-Linux-g++334
| |
| |-
| |
| | DropSite ||rowspan=4| ''DropSite'', ''DropLocation'', ''DropMethod'', and ''TriggerSite'' describe the location where test results are submitted. For more information about CTest submission, please look at: [[CTest:Submission Issues]]. The location specified in the example is the [http://public.kitware.com/dashboard.php?name=public Public Dashboard] provided as a service by [http://www.kitware.com Kitware]. || public.kitware.com
| |
| |-
| |
| | DropLocation|| /cgi-bin/HTTPUploadDartFile.cgi
| |
| |-
| |
| | DropMethod || http
| |
| |-
| |
| | TriggerSite || http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.pl
| |
| |-
| |
| | NightlyStartTime ||A the time when Nightly dashboard starts. This is useful when doing Nightly dashboard. The Nightly dashboard will update the source directory to the state at given time. || 21:00:00 EDT
| |
| |-
| |
| | UpdateCommand || Command used to update the project during the Update stage. || /usr/bin/cvs
| |
| |-
| |
| | UpdateOptions || Additional options passed to update command || -d -A -P
| |
| |-
| |
| | UpdateType || Type of update used. Currently only CVS and SVN are valid types || cvs
| |
| |-
| |
| | ConfigureCommand || Command run during the Configure stage || "/path/to/source/directory/Project/configure"
| |
| |-
| |
| | MakeCommand || Command run during the Build stage ||/usr/bin/make -i
| |
| |-
| |
| | CoverageCommand || Command used for the Coverage testing. Currently only GCC gcov is supported. || /usr/bin/gcov
| |
| |-
| |
| | TimeOut || A maximum time the test will be allowed to run. Once test time expires, the test will be interrupted and the status will be ''Timeout''. ||1500
| |
| |}
| |
| | |
| ===Description of Tests===
| |
| | |
| For legacy purposes and for compatibility with ''Dart'', the test description
| |
| file is called ''DartTestfile.txt''. It is parsed by full CMake parser, but the
| |
| only CMake commands that are available are ''SUBDIRS'' and ''ADD_TEST''. Example
| |
| ''DartTestfile.txt'' is:
| |
| | |
| <pre>
| |
| ADD_TEST(testProcess-1 "/my/project/testProcess" "1")
| |
| ADD_TEST(testProcess-2 "/my/project/testProcess" "2")
| |
| ADD_TEST(testProcess-3 "/my/project/testProcess" "3")
| |
| ADD_TEST(testProcess-4 "/my/project/testProcess" "4")
| |
| ADD_TEST(testProcess-5 "/my/project/testProcess" "5")
| |
| ADD_TEST(testProcess-6 "/my/project/testProcess" "6")
| |
| </pre>
| |
| | |
| Each of the ''ADD_TEST'' adds the test for CTest. The first argument is the
| |
| unique name of test. It should be unique, so that it can be differentiated
| |
| among all results. The second argument is the executable run as a part of the
| |
| test. This is followed by arbitrary number of arguments for the executable.
| |
| Executable can be build as a part of the project, or system executable can be
| |
| used. For example, to test a ''Python'' feature:
| |
| | |
| <pre>
| |
| ADD_TEST(testTkInter "/usr/bin/python" "-c" "import Tkinter")
| |
| </pre>
| |
| | |
| It is a good practice to specify full path to the executable name, since that
| |
| way CTest will always pick the desired executable. For example, let say the
| |
| test executable name is ''python'' but there is a python installed on the
| |
| system, the CTest may pick either the ''python'' executable built by our build
| |
| process or use the installed one.
| |
| | |
| ===Group Tests in Subdirectories===
| |
| | |
| Project tests are usually organized into some structure based on the part of the project they test. For example, if the project contains some common code, IO code, processing code, and GUI code, there may be group of tests for each subsection. Common way of organizing this would be in subdirectories. For example, let say the project has the following directory structure:
| |
| | |
| <pre>
| |
| /Project
| |
| /Common
| |
| /IO
| |
| /Processing
| |
| /GUI
| |
| </pre>
| |
| | |
| It is fairly easy to reproduce this structure for testing:
| |
| | |
| <pre>
| |
| /Project
| |
| /Testing
| |
| /Common
| |
| /IO
| |
| /Processing
| |
| /GUI
| |
| </pre>
| |
| | |
| CTest will not automatically search all subdirectories and find all
| |
| ''DartTestfile.txt''. The project needs to tell it where the tests are. This
| |
| can be achieved using ''SUBDIRS'' command in ''DartTestfile.txt''. An example use of ''SUBDIRS'' command is:
| |
| | |
| <pre>
| |
| SUBDIRS(Testing/Common)
| |
| SUBDIRS(Testing/IO)
| |
| SUBDIRS(Testing/Processing Testing/GUI)
| |
| </pre>
| |
| | |
| The arguments to the ''SUBDIRS'' command are subdirectories containing the
| |
| tests. If this file is placed in the top of our project, CTest will search for
| |
| files ''DartTestfile.txt'' inside directories /.../Project/Testing/Common,
| |
| /.../Project/Testing/IO, and so on to /.../Project/Testing/GUI.
| |
| | |
| ==Generating Testing Files==
| |
| | |
| Testing files ''DartConfiguration.tcl'' and all ''DartTestfile.txt' must be
| |
| generated in the same tree as the CTest is run. In most cases this is the build
| |
| tree of the project. Since the testing files contain build and system specific
| |
| information, they should be generated as a part of testing process and not
| |
| permanently part of source code.
| |
| | |
| ===Using CMake===
| |
| | |
| CMake has a automated way of generating CTest input files. There are three pieces to performing testing:
| |
| | |
| ENABLE_TESTING()
| |
| | |
| Enables testing for the project. This command will gather all tests and produce
| |
| ''DartTestfile.txt'' files with apropriate ''ADD_TEST'' and ''SUBDIRS''
| |
| commands.
| |
| | |
| INCLUDE(Dart)
| |
| | |
| Including ''Dart'' CMake module will generate default ''DartConfiguration.tcl''
| |
| file, which will contain information about updating, configuring, and building
| |
| the project.
| |
| | |
| ADD_TEST(test1 ${EXECUTABLE_OUTPUT_PATH}/test1)
| |
| | |
| Describes test that will be put in ''DartTestfile.txt''. The syntax of ADD_TEST on CMake level is the same as once it is written in ''DartTestfile.txt''.
| |
| | |
| More details about how to generate test files with CMake is described in
| |
| [[CMake Testing With CTest|Testing With CTest]] tutorial.
| |
| | |
| ===Without Using CMake===
| |
| | |
| CTest is distributed together with CMake, but it can be used independently. The
| |
| ''DartConfiguration.tcl'' file can be easily generated using ''autoconf'' or
| |
| any other tool, or even written by hand. Whatever the method of generating that
| |
| file, the author has to make sure all the necessary information is written in
| |
| it. Same goes for ''DartTestfile.txt'' files.
| |
| | |
| ==Conclusion==
| |
| | |
| Though we consider CMake to be a great build configuration tool, CTest can be
| |
| used without CMake. Because of its simplicity, it can be added to any project
| |
| without changing the rest of the project development. Using a small number of
| |
| configuration files, CTest can easily automate most if not all of the commonly
| |
| used testing procedures.
| |
| | |
| {{CMake/Template/Footer}}
| |
