[CMake] CTest examples

[Ryan Pavlik]

  I'm been using Boost Test, and I've written a module that lets you add 
the executable and the tests (where possible, individual named tests) 
with a single command - see my github repositories for projects that 
include this module.  http://github.com/rpavlik

On 7/9/10 9:04 PM, Alok Govil wrote:
> While I was suspecting something like this about CTest, your 
> explanation helps a lot.
>
> I have integrated UnitTest++ into my flow, and it is working quite 
> well.  It does also return a 0 if all tests pass, so one could 
> integrate with CTest/CDash if desirable.
>
> Thanks
> ------------------------------------------------------------------------
> Date: Fri, 9 Jul 2010 14:54:46 -0700
> From: chillery-cmake at lambda.nu
> To: bo at askmonty.org
> CC: cmake at cmake.org
> Subject: Re: [CMake] CTest examples
>
> For CTest, running a test means running an executable and seeing 
> whether the result of that process is 0 (success) or not 0 (failure). 
> It is also possible to scan the output of the executable for certain 
> regular expressions to determine success or failure.
>
> It's a very simple and somewhat limited mechanism. It does depend on 
> you writing your own test driver, which makes sense because CMake 
> can't possibly know how to test your code.
>
> As an example, Zorba is an XQuery (xml query language) processor. The 
> majority of our tests are actual XQueries stored in text files with 
> the extension .xq, along with corresponding expected results in .xml 
> files. We have a testdriver program which executes the query, checks 
> the results, and returns 0 or 1.
>
> Our CMakeLists looks vaguely like this:
>
> add_executable(testdriver ${TESTDRIVER_SRCS})
> file(GLOB_RECURSE TESTFILES "${CMAKE_CURRENT_SOURCE_DIR}" "*.xq")
> foreach(TESTFILE ${TESTFILES})
>   add_test(${TESTFILE} testdriver ${TESTFILE})
> endforeach()
>
> This creates one test for each .xq file. The name of the test is the 
> name of the .xq file, and that same name is passed as an argument to 
> testdriver.
>
> This works well, and lets us use ctest's functionality like -R to run 
> certain tests by name. It also results in nice easy-to-read listings 
> on CDash because there's one line for each test, making it easy to 
> examine failures. (Although given that we have over 20,000 tests, some 
> pages on CDash take a looooong time to load...) The main downside is 
> speed, since every single test has to start up a whole Zorba process.
>
> When you get to unit testing, however, IMHO things don't work as well. 
> CMake does offer the create_test_sourcelist() command, and you should 
> look into it. We use that in a couple places to create a single test 
> driver from several separate small unit test drivers. However, you 
> still need to write a loop in CMakeLists to add multiple tests using 
> that test driver.
>
> In another project I work on, I've been using CxxTest for a unit 
> testing framework. This is a really nice setup as it makes it very 
> trivial to write many unit tests. However, the integration with ctest 
> is poor. There's a reasonably good FindCxxTest script shipped with 
> CMake that makes it easy to get CxxTest suites running; in a nutshell:
>
> find_package(CxxTest)
> cxx_add_test(my_unit_test my_unit_test.cpp 
> "${CMAKE_CURRENT_SOURCE_DIR}/my_unit_test.h")
>
> Unit tests in CxxTest are written as classes in a .h file; 
> cxx_add_test() runs CxxTest to generate the named .cpp file from that, 
> then builds it into the named test driver and adds a CTest test for 
> it. So far so good. However, the main benefit of CxxTest is grouping 
> similar test cases into a single .h file, probably with similar setup 
> and so forth. Unfortunately cxx_add_test() only adds one test to 
> CTest, and that one test runs all the test cases in the .h file. So 
> you lose the ability to run certain tests with ctest -R, and the 
> reports on CDash will just show whether ALL the tests in my_unit_test 
> succeeded or not.
>
> The fundamental problem is that ctest requires a test case to be a 
> program execution, and has no facilities to allow a program to 
> represent multiple tests. So either you pay the cost in speed, or you 
> lose some of the flexibility ctest and cdash offer. (To be fair, even 
> if this were fixed in ctest, CxxTest would also need some 
> modifications as it currently doesn't let you run subsets of tests 
> either.)
>
> Ceej
> aka Chris Hillery
>
> On Fri, Jul 9, 2010 at 4:37 AM, Bo Thorsen <bo at askmonty.org 
> <mailto:bo at askmonty.org>> wrote:
>
>     Hi people,
>
>     I have converted a set of applications to cmake and cpack, and now
>     have my eyes set on ctest.
>
>     I'd like to hear if someone here has some good advice, or links to
>     good advice, on how to structure tests. I'm searching for help on
>     how to put different tests into what executables. On how to handle
>     multiple tests on each classes, on how to best structure the test
>     of the static libraries (all of those are part of the source tree)
>     that are linked in to the application. And on how to test classes
>     from the main application itself.
>
>     I have read the ctest FAQ, documentation etc. and still don't know
>     anything that help me write the actual test code.
>
>     From the looks of it, ctest only provides the framework to run a
>     test, no help is given to write the code of the tests themselves,
>     is this right? I have previously been using cppunit, and it looks
>     like this will still be useful.
>
>     To sum it up, I'm looking for real life advice on what you guys
>     have done with ctest. This information seem almost completely
>     missing on the net, where all searches on ctest leads to useless
>     presentation on ctest features.
>
>     Cheers,
>
>     Bo Thorsen.
>     Monty Program AB.
>
>     -- 
>
>     MariaDB: MySQL replacement
>     Community developed. Feature enhanced. Backward compatible.
>
>     _______________________________________________
>     Powered by www.kitware.com <http://www.kitware.com>
>
>     Visit other Kitware open-source projects at
>     http://www.kitware.com/opensource/opensource.html
>
>     Please keep messages on-topic and check the CMake FAQ at:
>     http://www.cmake.org/Wiki/CMake_FAQ
>
>     Follow this link to subscribe/unsubscribe:
>     http://www.cmake.org/mailman/listinfo/cmake
>
>
>
> _______________________________________________ Powered by 
> www.kitware.com Visit other Kitware open-source projects at 
> http://www.kitware.com/opensource/opensource.html Please keep messages 
> on-topic and check the CMake FAQ at: 
> http://www.cmake.org/Wiki/CMake_FAQ Follow this link to 
> subscribe/unsubscribe: http://www.cmake.org/mailman/listinfo/cmake
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake


-- 
Ryan Pavlik
Human-Computer Interaction Graduate Student
Virtual Reality Applications Center
Iowa State University

rpavlik at iastate.edu
http://academic.cleardefinition.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20100712/ee9bc152/attachment.htm>