CMake Generating Testing Files
Introduction
CTest is a powerful testing client compatible with Dart Dashboard system. It is distributed as a part of CMake but can be used with or without CMake. This tutorial explains some background behind the testing with CTest and the process of generating CTest testing files.
CTest Overview
CTest currently uses the old Dart input configuration file. New input format is being developed, but the old format will be valid in the future. For legacy purposes, the input files are called DartConfiguration.tcl, and DartTestfile.txt. DartConfiguration.tcl describes the system on which tests are performed, while DartTestfile.txt provides the list of tests.
CTest reads DartConfiguration.tcl and depending on the options provided it performs certain stage of testing.
Description Of The System
Even though the extension of DartConfiguration.tcl file is .tcl, it is not TCL file. It is simply list of key/value pairs. Line starting with # are considered comments and are ignored. This file has to appear at the toplevel directory of the project's build directory.
Example DartConfiguration.tcl file is:
SourceDirectory: /path/to/project/source/directory BuildDirectory: /path/to/project/build/directory CVSUpdateOptions: -d -A -P Site: my-system-name BuildName: build-name DropSite: public.kitware.com DropLocation: /cgi-bin/HTTPUploadDartFile.cgi DropMethod: http TriggerSite: http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.pl NightlyStartTime: 21:00:00 EDT ConfigureCommand: "/path/to/source/directory/Project/configure" MakeCommand: /usr/bin/make -i CVSCommand: /usr/bin/cvs CoverageCommand: /usr/bin/gcov TimeOut: 1500
DartConfiguration.tcl options
| Key | Description | Example value |
|---|---|---|
| SourceDirectory | Path to the source directory of the project. Used for CVS or SVN update. | /path/to/project/source/directory |
| BuildDirectory | Path to the build directory of the project. This is where Testing results will go and all testing commands will be run in this directory. | /path/to/project/build/directory |
| UpdateCommand | Command used to update the project. | /usr/bin/cvs |
| UpdateOptions | Additional options passed to update command | -d -A -P |
| UpdateType | Type of update used. Currently only CVS and SVN are valid types | cvs |
| Site | Refers to the system name used to distinguish the dashboard entry. | my-system-name |
| BuildName | A breif description of the system and the project. | FLTK-Linux-g++334 |
| DropSite | DropSite, DropLocation, DropMethod, and TriggerSite describe the location where test results are submitted. For more information about CTest submission, please look at: CTest:Submission Issues. The location specified in the example is the Public Dashboard provided as a service by Kitware. | public.kitware.com |
| DropLocation | /cgi-bin/HTTPUploadDartFile.cgi | |
| DropMethod | http | |
| TriggerSite | http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.pl | |
| NightlyStartTime | A the time when Nightly dashboard starts. This is useful when doing Nightly dashboard. The Nightly dashboard will update the source directory to the state at given time. | 21:00:00 EDT |
| ConfigureCommand | "/path/to/source/directory/Project/configure" | |
| MakeCommand | /usr/bin/make -i | |
| CVSCommand | /usr/bin/cvs | |
| CoverageCommand | /usr/bin/gcov | |
| TimeOut | A maximum time the test will be allowed to run. Once test time expires, the test will be interrupted and the status will be Timeout. | 1500 |
Description of Tests
For legacy purposes and for compatibility with Dart, the test description file is called DartTestfile.txt. It is parsed by full CMake parser, but the CMake commands that are available are SUBDIRS and ADD_TEST. Example DartTestfile.txt is:
ADD_TEST(testProcess-1 "/my/project/testProcess" "1") ADD_TEST(testProcess-2 "/my/project/testProcess" "2") ADD_TEST(testProcess-3 "/my/project/testProcess" "3") ADD_TEST(testProcess-4 "/my/project/testProcess" "4") ADD_TEST(testProcess-5 "/my/project/testProcess" "5") ADD_TEST(testProcess-6 "/my/project/testProcess" "6")
Each of the ADD_TEST adds the test for CTest. The first argument is the unique name of test. It should be unique, so that it can be differentiated among all results. The second argument is the executable run as a part of the test. This is followed by arbitrary number of arguments for the executable. Executable can be build as a part of the project, or system executable can be used. For example, to test a Python feature:
ADD_TEST(testTkInter "/usr/bin/python" "-c" "import Tkinter")
It is a good practice to specify full path to the executable name, since that way CTest will always pick the desired executable. For example, let say the test executable name is python but there is a python installed on the system, the CTest may pick either the python executable built by our build process or use the installed one.
Group Tests in Subdirectories
Project tests are usually organized into some structure based on the part of the project they test. For example, if the project contains some common code, IO code, processing code, and GUI code, there may be group of tests for each subsection. Common way of organizing this would be in subdirectories. For example, let say the project has the following directory structure:
/Project
/Common
/IO
/Processing
/GUI
It is fairly easy to reproduce this structure for testing:
/Project
/Testing
/Common
/IO
/Processing
/GUI
CTest will not automatically search all subdirectories and find all DartTestfile.txt. The project needs to tell it where the tests are. This can be achieved using SUBDIRS command in DartTestfile.txt. An example use of SUBDIRS command is:
SUBDIRS(Testing/Common) SUBDIRS(Testing/IO) SUBDIRS(Testing/Processing Testing/GUI)
The arguments to the SUBDIRS command are subdirectories containing the tests. If this file is placed in the top of our project, CTest will search for files DartTestfile.txt inside directories /.../Project/Testing/Common, /.../Project/Testing/IO, and so on to /.../Project/Testing/GUI.
Generating Testing Files
Testing files DartConfiguration.tcl and all DartTestfile.txt' must be generated in the same tree as the CTest is run. In most cases this is the build tree of the project. Since the testing files contain build and system specific information, they should be generated as a part of testing process and not permanently part of source code.
Using CMake
CMake has a automated way of generating CTest input files. There are three pieces to performing testing:
ENABLE_TESTING()
Enables testing for the project. This command will gather all tests and produce DartTestfile.txt files with apropriate ADD_TEST and SUBDIRS commands.
INCLUDE(Dart)
Including Dart CMake module will generate default DartConfiguration.tcl file, which will contain information about updating, configuring, and building the project.
ADD_TEST(test1 ${EXECUTABLE_OUTPUT_PATH}/test1)
Describes test that will be put in DartTestfile.txt. The syntax of ADD_TEST on CMake level is the same as once it is written in DartTestfile.txt.
More details about how to generate test files with CMake is described in Testing With CTest tutorial.
Without Using CMake
CTest is distributed together with CMake, but it can be used independently. The DartConfiguration.tcl file can be easily generated using autoconf or any other tool, or even written by hand. Whatever the method of generating that file, the author has to make sure all the necessary information is written in it. Same goes for DartTestfile.txt files.
Conclusion
Though we consider CMake to be a great build configuration tool, CTest can be used without CMake. Because of its simplicity, it can be added to any project without changing the rest of the project development. Using a small number of configuration files, CTest can easily automate most if not all of the commonly used testing procedures.
