[CMake] CTest + Catch Framework

[David Cole]

I have two more suggestions for whoever takes on this work.

One: add a new CMake command add_unit_test_container (or
add_unit_tester or a better name ... ?) which names the DLL or
executable file containing the unit tests. Give it properties to
indicate how to extract a list of tests from it (perhaps just a "type"
that maps to internal code that knows how to do this for each
type...), and how to run *all* the tests, and how to run an individual
test by name.

Two: consider performance, both of running all the unit tests and of
submitting/processing results on CDash, ... frequently, some unit test
containers involve "expensive" setup and teardown, and that's why it's
a good thing to have many tests in one container -- so the slow stuff
only has to run once for 100 tests... (or more!)

Unless you really really need this functionality, you might want to
test out CDash first to make sure it can handle the number of tests
you want to send it. Done properly, unit tests will test every single
C++ method in your code... If VTK were to do that, there would be
**tens of thousands** of test results being sent to CDash for each row
on the dashboard.

OK, and [Three:], make it a goal to add support for all of the
following (and make it easily extensible for more):

- gtest
- cppunit
- Catch
- Boost.Test
- Microsoft::VisualStudio::CppUnitTestFramework

I am currently involved with a project where we're using
Microsoft::VisualStudio::CppUnitTestFramework for unit tests, and we
run vstest.console.exe to run all the unit tests at once for each
library we test. Then we submit to a CDash dashboard as
"Release.UnitTests.LibraryName" and "Debug.UnitTests.LibraryName" --
if a single test fails, you just have to inspect the output and read
which test failed.

Unit tests, in general, should run nearly instantaneously, and ALL
your developers should run them and verify passing unit tests before
every push to the repo. If they're not doing that already, then new
ctest features ain't gonna help you...


Cheers,
David C.

On Wed, Jan 28, 2015 at 4:19 AM, Nils Gladitz <nilsgladitz at gmail.com> wrote:
> On 28.01.2015 01:38, Robert Dailey wrote:
>>
>> I suspect that per David's suggestion, CTest would essentially do what
>> you're doing but in a more thorough manner (based on the test
>> framework used). So essentially CTest will need:
>>
>> - A new command line parameter that specifies the test framework
>> - Functionality to parse the contents of source files provided to the
>> target the test is assigned to
>>
>> I'll try to look into this when I can. David, if you have more
>> specific requirements or pointers please do share them here. Thanks
>> for the input everyone.
>>
>
> CMake generates "CTestTestfile.cmake" files which contain the code that
> defines tests during execution of ctest itself.
>
> Those "CTestTestfile.cmake" files could use execute_process() to query test
> binaries for their contained tests and then add tests appropriately.
> With that approach there would be no need to parse source files.
>
> This can already be done in a hackish way by using the TEST_INCLUDE_FILE[1]
> directory property.
> file(GENERATE) can be used to create per-test-binary/per-configuration
> snippets with executable locations ($<TARGET_FILE>) which then can be
> globbed for and included by CTestTestfile.cmake.
>
> I would still opt for implementing support for this in cmake itself though
> since the entire logic could be contained in the generated
> "CTestTestfile.cmake"s.
> Perhaps with a couple of properties that would define how to e.g. query the
> list of tests and the required parameter to run a specific test; that way it
> could be made to work irregardless of which test framework is being used.
>
> Nils
>
> [1] http://www.cmake.org/cmake/help/v3.1/prop_dir/TEST_INCLUDE_FILE.html
>
> --
>
> Powered by www.kitware.com
>
> Please keep messages on-topic and check the CMake FAQ at:
> http://www.cmake.org/Wiki/CMake_FAQ
>
> Kitware offers various services to support the CMake community. For more
> information on each offering, please visit:
>
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Follow this link to subscribe/unsubscribe:
> http://public.kitware.com/mailman/listinfo/cmake