[CMake] CTest examples

[Alok Govil]

While I was suspecting something like this about CTest, your explanation helps a lot.

I have integrated UnitTest++ into my flow, and it is working quite well.  It does also return a 0 if all tests pass, so one could integrate with CTest/CDash if desirable.

Thanks
Date: Fri, 9 Jul 2010 14:54:46 -0700
From: chillery-cmake at lambda.nu
To: bo at askmonty.org
CC: cmake at cmake.org
Subject: Re: [CMake] CTest examples

For CTest, running a test means running an executable and seeing whether the result of that process is 0 (success) or not 0 (failure). It is also possible to scan the output of the executable for certain regular expressions to determine success or failure.


It's a very simple and somewhat limited mechanism. It does depend on you writing your own test driver, which makes sense because CMake can't possibly know how to test your code.

As an example, Zorba is an XQuery (xml query language) processor. The majority of our tests are actual XQueries stored in text files with the extension .xq, along with corresponding expected results in .xml files. We have a testdriver program which executes the query, checks the results, and returns 0 or 1.


Our CMakeLists looks vaguely like this:

add_executable(testdriver ${TESTDRIVER_SRCS})
file(GLOB_RECURSE TESTFILES "${CMAKE_CURRENT_SOURCE_DIR}" "*.xq")
foreach(TESTFILE ${TESTFILES})

  add_test(${TESTFILE} testdriver ${TESTFILE})
endforeach()

This creates one test for each .xq file. The name of the test is the name of the .xq file, and that same name is passed as an argument to testdriver.


This works well, and lets us use ctest's functionality like -R to run certain tests by name. It also results in nice easy-to-read listings on CDash because there's one line for each test, making it easy to examine failures. (Although given that we have over 20,000 tests, some pages on CDash take a looooong time to load...) The main downside is speed, since every single test has to start up a whole Zorba process.


When you get to unit testing, however, IMHO things don't work as well. CMake does offer the create_test_sourcelist() command, and you should look into it. We use that in a couple places to create a single test driver from several separate small unit test drivers. However, you still need to write a loop in CMakeLists to add multiple tests using that test driver.


In another project I work on, I've been using CxxTest for a unit testing framework. This is a really nice setup as it makes it very trivial to write many unit tests. However, the integration with ctest is poor. There's a reasonably good FindCxxTest script shipped with CMake that makes it easy to get CxxTest suites running; in a nutshell:


find_package(CxxTest)
cxx_add_test(my_unit_test my_unit_test.cpp "${CMAKE_CURRENT_SOURCE_DIR}/my_unit_test.h")

Unit tests in CxxTest are written as classes in a .h file; cxx_add_test() runs CxxTest to generate the named .cpp file from that, then builds it into the named test driver and adds a CTest test for it. So far so good. However, the main benefit of CxxTest is grouping similar test cases into a single .h file, probably with similar setup and so forth. Unfortunately cxx_add_test() only adds one test to CTest, and that one test runs all the test cases in the .h file. So you lose the ability to run certain tests with ctest -R, and the reports on CDash will just show whether ALL the tests in my_unit_test succeeded or not.


The fundamental problem is that ctest requires a test case to be a program execution, and has no facilities to allow a program to represent multiple tests. So either you pay the cost in speed, or you lose some of the flexibility ctest and cdash offer. (To be fair, even if this were fixed in ctest, CxxTest would also need some modifications as it currently doesn't let you run subsets of tests either.)


Ceej
aka Chris Hillery

On Fri, Jul 9, 2010 at 4:37 AM, Bo Thorsen <bo at askmonty.org> wrote:

Hi people,



I have converted a set of applications to cmake and cpack, and now have my eyes set on ctest.



I'd like to hear if someone here has some good advice, or links to good advice, on how to structure tests. I'm searching for help on how to put different tests into what executables. On how to handle multiple tests on each classes, on how to best structure the test of the static libraries (all of those are part of the source tree) that are linked in to the application. And on how to test classes from the main application itself.




I have read the ctest FAQ, documentation etc. and still don't know anything that help me write the actual test code.