[CMake] CTest + Catch Framework

[Robert Dailey]

@David

I agree with your point about unit test initialization. Perhaps
instead of running a test multiple times, we can inspect the output of
a test that is run 1 time and assuming it spits out success/failure
details for individual test cases, we can simply parse that output
into appropriate CText XML so that it appears as if it was instead run
once for each test case?

I'm being fairly abstract here as I'm not sure of the implementation
details of CTest or if this is even possible. But it's a thought. Let
me know what you think.

On Wed, Jan 28, 2015 at 6:43 AM, David Cole <DLRdave at aol.com> wrote:
> I have two more suggestions for whoever takes on this work.
>
> One: add a new CMake command add_unit_test_container (or
> add_unit_tester or a better name ... ?) which names the DLL or
> executable file containing the unit tests. Give it properties to
> indicate how to extract a list of tests from it (perhaps just a "type"
> that maps to internal code that knows how to do this for each
> type...), and how to run *all* the tests, and how to run an individual
> test by name.
>
> Two: consider performance, both of running all the unit tests and of
> submitting/processing results on CDash, ... frequently, some unit test
> containers involve "expensive" setup and teardown, and that's why it's
> a good thing to have many tests in one container -- so the slow stuff
> only has to run once for 100 tests... (or more!)
>
> Unless you really really need this functionality, you might want to
> test out CDash first to make sure it can handle the number of tests
> you want to send it. Done properly, unit tests will test every single
> C++ method in your code... If VTK were to do that, there would be
> **tens of thousands** of test results being sent to CDash for each row
> on the dashboard.
>
> OK, and [Three:], make it a goal to add support for all of the
> following (and make it easily extensible for more):
>
> - gtest
> - cppunit
> - Catch
> - Boost.Test
> - Microsoft::VisualStudio::CppUnitTestFramework
>
> I am currently involved with a project where we're using
> Microsoft::VisualStudio::CppUnitTestFramework for unit tests, and we
> run vstest.console.exe to run all the unit tests at once for each
> library we test. Then we submit to a CDash dashboard as
> "Release.UnitTests.LibraryName" and "Debug.UnitTests.LibraryName" --
> if a single test fails, you just have to inspect the output and read
> which test failed.
>
> Unit tests, in general, should run nearly instantaneously, and ALL
> your developers should run them and verify passing unit tests before
> every push to the repo. If they're not doing that already, then new
> ctest features ain't gonna help you...
>
>
> Cheers,
> David C.
>
> On Wed, Jan 28, 2015 at 4:19 AM, Nils Gladitz <nilsgladitz at gmail.com> wrote:
>> On 28.01.2015 01:38, Robert Dailey wrote:
>>>
>>> I suspect that per David's suggestion, CTest would essentially do what
>>> you're doing but in a more thorough manner (based on the test
>>> framework used). So essentially CTest will need:
>>>
>>> - A new command line parameter that specifies the test framework
>>> - Functionality to parse the contents of source files provided to the
>>> target the test is assigned to
>>>
>>> I'll try to look into this when I can. David, if you have more
>>> specific requirements or pointers please do share them here. Thanks
>>> for the input everyone.
>>>
>>
>> CMake generates "CTestTestfile.cmake" files which contain the code that
>> defines tests during execution of ctest itself.
>>
>> Those "CTestTestfile.cmake" files could use execute_process() to query test
>> binaries for their contained tests and then add tests appropriately.
>> With that approach there would be no need to parse source files.
>>
>> This can already be done in a hackish way by using the TEST_INCLUDE_FILE[1]
>> directory property.
>> file(GENERATE) can be used to create per-test-binary/per-configuration
>> snippets with executable locations ($<TARGET_FILE>) which then can be
>> globbed for and included by CTestTestfile.cmake.
>>
>> I would still opt for implementing support for this in cmake itself though
>> since the entire logic could be contained in the generated
>> "CTestTestfile.cmake"s.
>> Perhaps with a couple of properties that would define how to e.g. query the
>> list of tests and the required parameter to run a specific test; that way it
>> could be made to work irregardless of which test framework is being used.
>>
>> Nils
>>
>> [1] http://www.cmake.org/cmake/help/v3.1/prop_dir/TEST_INCLUDE_FILE.html
>>
>> --
>>
>> Powered by www.kitware.com
>>
>> Please keep messages on-topic and check the CMake FAQ at:
>> http://www.cmake.org/Wiki/CMake_FAQ
>>
>> Kitware offers various services to support the CMake community. For more
>> information on each offering, please visit:
>>
>> CMake Support: http://cmake.org/cmake/help/support.html
>> CMake Consulting: http://cmake.org/cmake/help/consulting.html
>> CMake Training Courses: http://cmake.org/cmake/help/training.html
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Follow this link to subscribe/unsubscribe:
>> http://public.kitware.com/mailman/listinfo/cmake