[CMake] Multiple tests in a single file

[David Cole]

On Sunday, January 8, 2012, Fraser Hutchison <
fraser.hutchison at googlemail.com> wrote:
> On 07/01/2012 14:41, David Cole wrote:
>
> On Fri, Jan 6, 2012 at 6:47 PM, David Doria <daviddoria at gmail.com> wrote:
>
> On Fri, Jan 6, 2012 at 5:54 PM, Jean-Christophe Fillion-Robin
> <jchris.fillionr at kitware.com> wrote:
>
> Hi David,
>
> Not too long ago I was browsing the project of a friend who worked on BTK
> (The toolkit used by Mokka, a motion kinematic & kinetic analyser) [1].
>
> I noticed that he is using cxxtest [2] along with ctest. It seems it could
> to address the use case you are describing.
>
> See Example integration [3] and cxxtest source [4]
>
> Hth
> Jc
>
> [1] https://b-tk.googlecode.com/svn/web/mokka/index.html
> [2] http://cxxtest.tigris.org/
> [3]
>
https://code.google.com/p/b-tk/source/browse/BTK/trunk/Testing/Code/C3DFileReaderTest.h
> [4]
>
https://code.google.com/p/b-tk/source/browse/#svn%2FBTK%2Ftrunk%2FUtilities%2FCxxTest%2Fcxxtest
>
> Thanks Jean-Christophe. So I guess the short answer is "ctest can't do
this".
>
> It seems they have made significant modifications to CxxTest so that
> the bin/cxxtestgen that ships with CxxTest is no longer required
> (they've replaced it with C macros). I like the BTK way better :)
>
> I guess I'm not sure what the advantage of using CTest along with
> something like CxxTest is. If you build an executable using CMake,
> then why not just run ./MyTests from the terminal instead of making an
> interface so that you can call 'ctest'. CTest will report a single
> pass or fail, because as far as it is concerned there is only one
> test, right? The relevant file is
>
https://code.google.com/p/b-tk/source/browse/BTK/trunk/Testing/Code/CMakeLists.txt
> that only has one add_test call.
>
> Any thoughts?
>
> David
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>
> Sounds like you want to use GTest / GoogleTest, much like SimpleITK does.
>
> See the macro "ADD_GOOGLE_TESTS" in the SimpleITK project, in the file
> SimpleITK/Testing/Unit/CMakeLists.txt.
>
> The macro parses the actual test cxx source files, and constructs
> "filtered" add_test calls, such that:
>
> - if a cxx file defines 5 tests, add_test is called 5 times, each test
> run via add_test runs exactly one of your tests from the cxx file.
>
> There are other projects that are also doing similar things with
> gtest. The fairly new open chemistry project MolCore also uses gtest.
> That project explicitly lists its tests in the CMakeLists file such
> that you'd have to update the CMakeLists.txt file whenever you add a
> test case to an existing cxx test source file.
>
> You are correct: ctest does nothing to support this explicitly. (Aside
> from cmake providing a FindGTest.cmake module...) An add_test call
> tells ctest "run this one test, with this one command line." It's up
> to you to write those tests and tell us about them.
>
> But there are fairly easy ways to do what you want, and still drive
> the tests, individually, with ctest.
>
>
> HTH,
> David C.
> --
>
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html
>
> Please keep messages on-topic and check the CMake FAQ at:
http://www.cmake.org/Wiki/CMake_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake
>
> We found a few issues using gtest in the way you describe David:
>
> Parsing C++ code isn't CMake's thing, so it becomes increasingly
difficult to support anything other than very simple gtest macros (like
TEST_P, TYPED_TEST, TYPED_TEST_P).  We do handle these macros now, but I'm
sure it's pretty fragile, and we don't even attempt to handle things like
C++ namespace rules within test files!
> gtest provides the ability to set up (potentially costly) test
environments which persist throughout execution of all tests, rather than
on a per-test basis.  Using this, and then having CTest invoke each test
individually totally defeats the purpose of this feature.
> Commenting out a test using block comments leaves the test name
unchanged, so CTest still tries to run it.
> If a gtest exe is run with a filter which doesn't match any test names,
it runs 0 tests and returns 0 which CTest interprets as success.  This
could happen if 3 above were the case.  We've got a policy of using the
following in main() to avoid reporting success:
>
>     int result(RUN_ALL_TESTS());
>     int test_count(testing::UnitTest::GetInstance()->test_to_run_count());
>     return (test_count == 0) ? -1 : result;
>
> gtest exes can be run with the flag --gtest_list_tests which outputs all
test names in that exe.  It would be great to use that to generate the list
of filters needed, but that suffers from needing the exe to exist before
CMake is run.  So if a new test case is added, it won't be run by CTest
until CMake is re-run.
>
> Having said all that, we continue to like gtest and are fairly used to
the limitations mentioned above.  Whatever problems we've hit, there's
always been a CTest way to get round the issue.
>
> Cheers,
>
> Fraser.
>
>

Interesting. Thx for chiming in.

Let us know if you have any suggestions or good ideas for how we could
improve things. Maybe a test property that identifies an exe as a gtest
exe, and then runs it to figure out the list of tests to run.


David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20120108/1e345346/attachment-0001.htm>