[CMake] CTest + Catch Framework

[Fraser Hutchison]

We've run into this exact chicken and egg problem with both gtest and 
Catch at my work too.  In both cases, we've worked round the problem by 
writing fairly fragile CMake code which parses the C++ files comprising 
a test executable and ultimately calling add_test for each discovered 
test case.

gtest is harder to handle than Catch in that it provides macros for 
handling value-parameterised and type-parameterised tests.  It would be 
great if CTest were able to handle these popular test libraries 
automatically.  I'm really short of time of the moment, but if there are 
any pointers from experts on how this might be best handled by CTest, I 
could perhaps start to have a poke about if there are no other takers?

In the meantime, here's a link to our Catch-handling code: 
https://github.com/Fraser999/MaidSafe/blob/846ce37c23bd86ee261e439919dfd7000a8f9372/cmake_modules/add_catch_tests.cmake

The file's commented at the top.  It probably doesn't cover all Catch 
macros, and I'd be amazed if it handled all C++ coding styles properly, 
but it worked fairly reliably for us.  We don't use Catch any more, so 
I'm not even sure if it works with the current version of Catch.  For 
comparison (of the complexity) here's our gtest equivalent: 
https://github.com/Fraser999/MaidSafe/blob/72af3a0def2f03af7320696f5ea3d241c5af9bdc/cmake_modules/add_gtests.cmake 
(it's old code and could be made more efficient I'm sure - but again it 
works fine for us).

Cheers,
Fraser.


On 27/01/2015 17:14, David Cole via CMake wrote:
> The best thing to do would be to make CTest aware of unit test
> frameworks like this, and have it be automatic.
>
> Support for cppunit, gtest and Catch (and any other popular ones)
> would all be appreciated by many, I'm sure.
>
> Got time to dig into ctest a little bit?
>
>
> D
>
>
>
> On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey
> <rcdailey.lists at gmail.com> wrote:
>> I wouldn't say this is an artificial problem. let me go into more detail.
>>
>> A single test can define any number of test cases. These are unknown
>> to CMake scripts, as they are written and defined in code. The ideal
>> solution is to have the test program output a list of all test cases
>> and based on that list, automatically generate the add_test commands.
>> That way I don't have to worry about hard-coding test case names in
>> the build scripts. When a new test case is added to the test code (no
>> new files would be added), "magically" a new unit test appears to
>> CMake.
>>
>> But none of this can be done unless the unit test executables are
>> built. Thus the chicken and egg problem. To do it as you suggest would
>> require me to hard-code the test cases. Example:
>>
>> In MyTest.cpp:
>>
>> #include <catch.hpp>
>> TEST_CASE("test1") {}
>> TEST_CASE("test2") {}
>>
>> In CMakeLists.txt:
>>
>> add_executable( MyTest MyTest.cpp )
>> add_test( NAME MyTest_test1 COMMAND test1 )
>> add_test( NAME MyTest_test2 COMMAND test2 )
>>
>> Am I correctly understanding what you have in mind? If so, can you
>> think of a way to automate this a little better?
>>
>> On Tue, Jan 27, 2015 at 7:18 AM, David Cole <DLRdave at aol.com> wrote:
>>> It's only a chicken and egg problem if you constrain yourself artificially.
>>>
>>> When you define a new unit test source file, there are two things you
>>> need to do:
>>> (1) add it to its executable
>>> (2) write an add_test line that invokes that specific unit test
>>>
>>> You could possibly encapsulate those actions into an add_unit_test
>>> macro so that it only seems like you have one thing to do, but
>>> certainly, it's possible.
>>>
>>> You must be doing #1 already, so you just have to make sure #2 always
>>> happens whenever #1 happens.
>>>
>>> Both eggs.
>>>
>>>
>>> D
>>>
>>>
>>>
>>>
>>> On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey <rcdailey.lists at gmail.com> wrote:
>>>> I believe so, but then you run into the chicken and egg problem:
>>>>
>>>> I need to fully build all unit tests in order to know all the test
>>>> cases, but I need to define the tests at the generation phase (before
>>>> build).
>>>>
>>>> I'm just not sure how to handle this. The best i can think of is to
>>>> generate 1 test executable per CPP file that we write in our unit
>>>> tests (generally 1 CPP file per class that is unit tested). Any ideas?
>>>>
>>>> On Mon, Jan 26, 2015 at 10:05 AM, David Cole <DLRdave at aol.com> wrote:
>>>>> Can you run a command that executes one test case at a time, or do you
>>>>> have to run all 50 when you execute the tests for a library?
>>>>>
>>>>> ctest just runs whatever you give it with add_test -- if you want to
>>>>> filter a collection of unit tests to run only a single unit test, then
>>>>> the unit test framework you're using would have to support that. Does
>>>>> Catch allow you to run the tests individually?
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey
>>>>> <rcdailey.lists at gmail.com> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I'm using Catch as my unit test framework:
>>>>>> https://github.com/philsquared/Catch
>>>>>>
>>>>>> Is it possible for CTest to report each TEST_CASE block as an
>>>>>> individual test? My understanding is that CTest will only treat each
>>>>>> executable as a test. However, my test structure is as follows:
>>>>>>
>>>>>> 1 library
>>>>>> 1 test project
>>>>>>
>>>>>> Each test project has 1 test CPP file per each class in the library.
>>>>>> This way I implement tests for classes in a corresponding CPP file.
>>>>>>
>>>>>> Each CPP file contains multiple test cases (defined by TEST_CASE macro).
>>>>>>
>>>>>> The resulting output of `ctest -T Test` shows only 1 test, even though
>>>>>> that may be around 50 test cases. I'd like CMake to show the pass/fail
>>>>>> status of each one. Is this possible?
>>>>>> --
>>>>>>
>>>>>> Powered by www.kitware.com
>>>>>>
>>>>>> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
>>>>>>
>>>>>> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
>>>>>>
>>>>>> CMake Support: http://cmake.org/cmake/help/support.html
>>>>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>>>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>>>>>
>>>>>> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
>>>>>>
>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>> http://public.kitware.com/mailman/listinfo/cmake