[CMake] CTest + Catch Framework

Tue Jan 27 19:38:25 EST 2015

I suspect that per David's suggestion, CTest would essentially do what
you're doing but in a more thorough manner (based on the test
framework used). So essentially CTest will need:

- A new command line parameter that specifies the test framework
- Functionality to parse the contents of source files provided to the
target the test is assigned to

I'll try to look into this when I can. David, if you have more
specific requirements or pointers please do share them here. Thanks
for the input everyone.

On Tue, Jan 27, 2015 at 5:08 PM, Fraser Hutchison
<fraser.hutchison at gmail.com> wrote:
> We've run into this exact chicken and egg problem with both gtest and Catch
> at my work too.  In both cases, we've worked round the problem by writing
> fairly fragile CMake code which parses the C++ files comprising a test
> executable and ultimately calling add_test for each discovered test case.
>
> gtest is harder to handle than Catch in that it provides macros for handling
> value-parameterised and type-parameterised tests.  It would be great if
> CTest were able to handle these popular test libraries automatically.  I'm
> really short of time of the moment, but if there are any pointers from
> experts on how this might be best handled by CTest, I could perhaps start to
> have a poke about if there are no other takers?
>
> In the meantime, here's a link to our Catch-handling code:
> https://github.com/Fraser999/MaidSafe/blob/846ce37c23bd86ee261e439919dfd7000a8f9372/cmake_modules/add_catch_tests.cmake
>
> The file's commented at the top.  It probably doesn't cover all Catch
> macros, and I'd be amazed if it handled all C++ coding styles properly, but
> it worked fairly reliably for us.  We don't use Catch any more, so I'm not
> even sure if it works with the current version of Catch.  For comparison (of
> the complexity) here's our gtest equivalent:
> https://github.com/Fraser999/MaidSafe/blob/72af3a0def2f03af7320696f5ea3d241c5af9bdc/cmake_modules/add_gtests.cmake
> (it's old code and could be made more efficient I'm sure - but again it
> works fine for us).
>
> Cheers,
> Fraser.
>
>
>
> On 27/01/2015 17:14, David Cole via CMake wrote:
>>
>> The best thing to do would be to make CTest aware of unit test
>> frameworks like this, and have it be automatic.
>>
>> Support for cppunit, gtest and Catch (and any other popular ones)
>> would all be appreciated by many, I'm sure.
>>
>> Got time to dig into ctest a little bit?
>>
>>
>> D
>>
>>
>>
>> On Tue, Jan 27, 2015 at 11:18 AM, Robert Dailey
>> <rcdailey.lists at gmail.com> wrote:
>>>
>>> I wouldn't say this is an artificial problem. let me go into more detail.
>>>
>>> A single test can define any number of test cases. These are unknown
>>> to CMake scripts, as they are written and defined in code. The ideal
>>> solution is to have the test program output a list of all test cases
>>> and based on that list, automatically generate the add_test commands.
>>> That way I don't have to worry about hard-coding test case names in
>>> the build scripts. When a new test case is added to the test code (no
>>> new files would be added), "magically" a new unit test appears to
>>> CMake.
>>>
>>> But none of this can be done unless the unit test executables are
>>> built. Thus the chicken and egg problem. To do it as you suggest would
>>> require me to hard-code the test cases. Example:
>>>
>>> In MyTest.cpp:
>>>
>>> #include <catch.hpp>
>>> TEST_CASE("test1") {}
>>> TEST_CASE("test2") {}
>>>
>>> In CMakeLists.txt:
>>>
>>> add_executable( MyTest MyTest.cpp )
>>> add_test( NAME MyTest_test1 COMMAND test1 )
>>> add_test( NAME MyTest_test2 COMMAND test2 )
>>>
>>> Am I correctly understanding what you have in mind? If so, can you
>>> think of a way to automate this a little better?
>>>
>>> On Tue, Jan 27, 2015 at 7:18 AM, David Cole <DLRdave at aol.com> wrote:
>>>>
>>>> It's only a chicken and egg problem if you constrain yourself
>>>> artificially.
>>>>
>>>> When you define a new unit test source file, there are two things you
>>>> need to do:
>>>> (1) add it to its executable
>>>> (2) write an add_test line that invokes that specific unit test
>>>>
>>>> You could possibly encapsulate those actions into an add_unit_test
>>>> macro so that it only seems like you have one thing to do, but
>>>> certainly, it's possible.
>>>>
>>>> You must be doing #1 already, so you just have to make sure #2 always
>>>> happens whenever #1 happens.
>>>>
>>>> Both eggs.
>>>>
>>>>
>>>> D
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Jan 26, 2015 at 4:05 PM, Robert Dailey
>>>> <rcdailey.lists at gmail.com> wrote:
>>>>>
>>>>> I believe so, but then you run into the chicken and egg problem:
>>>>>
>>>>> I need to fully build all unit tests in order to know all the test
>>>>> cases, but I need to define the tests at the generation phase (before
>>>>> build).
>>>>>
>>>>> I'm just not sure how to handle this. The best i can think of is to
>>>>> generate 1 test executable per CPP file that we write in our unit
>>>>> tests (generally 1 CPP file per class that is unit tested). Any ideas?
>>>>>
>>>>> On Mon, Jan 26, 2015 at 10:05 AM, David Cole <DLRdave at aol.com> wrote:
>>>>>>
>>>>>> Can you run a command that executes one test case at a time, or do you
>>>>>> have to run all 50 when you execute the tests for a library?
>>>>>>
>>>>>> ctest just runs whatever you give it with add_test -- if you want to
>>>>>> filter a collection of unit tests to run only a single unit test, then
>>>>>> the unit test framework you're using would have to support that. Does
>>>>>> Catch allow you to run the tests individually?
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 26, 2015 at 12:30 AM, Robert Dailey
>>>>>> <rcdailey.lists at gmail.com> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I'm using Catch as my unit test framework:
>>>>>>> https://github.com/philsquared/Catch
>>>>>>>
>>>>>>> Is it possible for CTest to report each TEST_CASE block as an
>>>>>>> individual test? My understanding is that CTest will only treat each
>>>>>>> executable as a test. However, my test structure is as follows:
>>>>>>>
>>>>>>> 1 library
>>>>>>> 1 test project
>>>>>>>
>>>>>>> Each test project has 1 test CPP file per each class in the library.
>>>>>>> This way I implement tests for classes in a corresponding CPP file.
>>>>>>>
>>>>>>> Each CPP file contains multiple test cases (defined by TEST_CASE
>>>>>>> macro).
>>>>>>>
>>>>>>> The resulting output of `ctest -T Test` shows only 1 test, even
>>>>>>> though
>>>>>>> that may be around 50 test cases. I'd like CMake to show the
>>>>>>> pass/fail
>>>>>>> status of each one. Is this possible?
>>>>>>> --
>>>>>>>
>>>>>>> Powered by www.kitware.com
>>>>>>>
>>>>>>> Please keep messages on-topic and check the CMake FAQ at:
>>>>>>> http://www.cmake.org/Wiki/CMake_FAQ
>>>>>>>
>>>>>>> Kitware offers various services to support the CMake community. For
>>>>>>> more information on each offering, please visit:
>>>>>>>
>>>>>>> CMake Support: http://cmake.org/cmake/help/support.html
>>>>>>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>>>>>>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>>>>>>
>>>>>>> Visit other Kitware open-source projects at
>>>>>>> http://www.kitware.com/opensource/opensource.html
>>>>>>>
>>>>>>> Follow this link to subscribe/unsubscribe:
>>>>>>> http://public.kitware.com/mailman/listinfo/cmake
>
>