[CMake] Building and executing tests when building updated libraries

[Eric Noulard]

2012/2/21 Robert Dailey <rcdailey at gmail.com>:
>
> I apologize if I sounded like your suggestion wasn't meaningful or useful. I
> would much rather prefer to do it how you suggest (running all tests), but
> this leaves me with some concerns:
>
> If the developer is running all unit tests on their local machine, what is
> the purpose of then running them on the server? If the server does it again
> in response to the commit, wouldn't that be considered redundant?

The server may catch interaction between modifications that make
some tests fails when features are merged but not when tested alone.

This would depend on you VCS as well. With a centralized VCS like
(CVS, Subversion etc...)
this would be odd if this happened. With DVCS with some
non-conflicting merge workflow
(think of the 'next' CMake branch) then server may reveal errors that
developers cannot catch.
Moreover in "portable" project like those I know using CMake the
server may in fact be
serversssZZ which may have very different setup than the one the
developper is running on.
(I do break MacOS X or Windows whatever version compile when I develop
on my Linux box).

The "server" breakage may reveal that the project is lacking some test
which would have ease the unitary catch.
Do you have 100% coverage with your tests?

> Let's assume that the time it takes to run all tests takes about 1 hour. Not
> only does this slow down productivity, but it also makes practices like
> continuous integration impossible to perform, since a lot of people can
> commit work in that 1 hour window, in which case you'd have to run the tests
> again after updating. It's a recursive issue.

If the "whole" test is lengthly may be the tested project is beginning
to be too big
to be tested as a whole. Imagine the test suite a for a whole
operating system...

Then the project may be subdivided into smaller "independent" projects, that
can be tested independently & safely in a shorter time.

Then you create some "integration project" (and put someone responsible for it)
which essentially consist integration tests which aggregate & use the
smaller projects.
This "integration test suite" may be run extensively before each
"combo release"
and/or periodically like once a week.

> How would you address the concerns I have noted above?

My 2 cents just above.
Plus ask some people with a huge number of tests like OTB:
http://dash.orfeo-toolbox.org/index.php?project=OTB&date=2012-02-21

Totals	5 Builds ... 10464 tests :-]

I bet they don't run the whole test suite each time they commit.

> My tests are labeled in such a way that they are easy to spot and work with
> in the solution explorer. For example:
>
> projectA
> projectA_test_iostreams
> projectA_test_fileio
> projectA_test_graphics
> projectA_test_input

Then you have a neat way to run ctest efficiently on those
using ctest -L  (or ctest -R if your are speaking of test names and
not "label").

By the way would someone explain me (or point me to appropriate place)
on test "label" usage, are those derived from labels that could be put on target
or may I put label on test as well (I know I can try, but sometimes I
like to read as well).

> In my example above, target named "projectA" has 4 unit tests. Each test can
> be responsible for 1 or more translation units (there is no strict rule
> here). If I change the way files are loaded by library "projectA", then I
> would run the fileio test. However, in this case it's really easy for the
> developer to spot the tests for that project and run all of them if they are
> unsure.
>
> Would you also mind commenting on this structure? It seems to ease the
> concern you mentioned about people not always being able to know which tests
> to run.
>
> Thanks for your feedback. Keep in mind that not only am I covering general
> testing principles, but I also want to know how to best apply them to the
> tools (CMake, CTest). This is where your expertise becomes valuable to me :)



-- 
Erk
Membre de l'April - « promouvoir et défendre le logiciel libre » -
http://www.april.org