[CMake] Building and executing tests when building updated libraries

[David Cole]

On Tue, Feb 21, 2012 at 1:51 PM, Robert Dailey <rcdailey at gmail.com> wrote:
> On Tue, Feb 21, 2012 at 12:37 PM, David Cole <david.cole at kitware.com> wrote:
>>
>> On Tue, Feb 21, 2012 at 1:27 PM, Robert Dailey <rcdailey at gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I'm using Visual Studio as my generator for my CMake projects. As of
>>> right now, I make my tests depend on the libraries they test. So for
>>> example, tests named:
>>>
>>> test_thingA
>>> test_thingB
>>>
>>> will all depend on library:
>>>
>>> libfoo.lib
>>>
>>> When I build target "libfoo" in visual studio, it would be nice to have
>>> all dependent tests build as well, and have them each execute.
>>>
>>> The goal for all of this is to make it as convenient as possible for
>>> developers on my team to RUN TESTS on their code before they submit to
>>> version control. I want to make it automated, so when they rebuild the
>>> library, the testing automatically happens. I'd also obviously create an
>>> option in cmake cache to turn this automation off should it become too
>>> annoying.
>>>
>>> If this isn't a good idea, can someone recommend a good workflow for
>>> running tests locally prior to checking in source code?
>>>
>>> ---------
>>> Robert Dailey
>>>
>>> --
>>>
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Please keep messages on-topic and check the CMake FAQ at:
>>> http://www.cmake.org/Wiki/CMake_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.cmake.org/mailman/listinfo/cmake
>>
>>
>>
>> If you're using add_test in your CMakeLists files, then the perfect way to
>> prove that the tests all work on a developer's machine is for him or her to
>> run:
>>
>>   ctest -D Experimental
>>
>> after making local mods, and before pushing the changes to your source
>> control system.
>>
>> That will configure, build all and run all the tests. And submit the
>> results to your CDash server so that is public evidence that he actually did
>> run the tests, and hopefully that they all passed on his machine at least.
>>
>> You can also restrict the set of tests that run using -R or -I or -L on
>> the ctest command line, although, you should strive to have your test suite
>> be brief enough that it's not painful for folks to run the full test suite
>> prior to checkin.
>
>
> I think this is a reasonable idea for small projects, but in general I
> disagree with running all tests.
>
> There are hundreds of projects (probably 150) and hundreds more of tests
> (probably 10 tests per project). In general agile methodology, it only makes
> sense to unit test those components which have changed. Unit testing a
> dependent component that did not have a source code change will not be
> needed or beneficial.
>
> All of these tests can take hours to run, which isn't unacceptable because
> it's a full test suite. Only the build server kicks off a build and runs the
> FULL test suite (thus running ctest -D Experimental as you have suggested).
> Developers just do an intermediate check by unit testing only the parts of
> the code base that have changed. This is essential for practices like
> continuous integration.
>
> Ideally the pipeline goes like this:
>
> Programmer makes a change to a certain number of libraries
> Programmer runs the relevant tests (or all) for each of the libraries that
> were changed.
> Once those tests have passed, the developer submits the source code to
> version control
> The build server is then instructed to run a full build and test of the
> entire code base for each checkin.
> The build server can then run any integration tests that are configured (not
> sure how these would be setup in CMake - probably again as tests, but not
> specific to only a single project)
> Build is considered "complete" at this point.
>
> Seems like there would be no choice but to run them individually in this
> case, since CMake really shines only in steps after #3

Incremental testing is something we've talked about over the years,
but there's no concept of "what's changed, what needs to run" when
ctest runs at the moment. Communicating that information from the
build to ctest, or making testing always part of the build are the two
approaches we've considered. Nothing exists yet, though, so it's all
in the future, yet to come.

Sorry to hear you disagree about running all the tests. I'll make one
more point and then shut up about it: the larger the project, the more
you need to run all the tests when changes are made. Unless the
developers all have a very good understanding of what parts need to be
tested when they make a change, they should run all the tests. If a
system is very large, then developers are more likely to have
imperfect understandings about the system... when that's the case, if
there is any doubt at all about what dependencies exist, then all the
tests should be run to verify a change.

Until incremental testing is available, I'd say your best bet is to
run all the tests.


HTH,
David