[ITK Community] [Insight-developers] Testing of Remote Modules

[Brian Helba]

This patch breaks the following use case:
* Module_A depends upon Module_B
* user disables BUILD_DEFAULT_MODULES, enables Module_A
* note that after a re-configure, the CMake option for Module_B has
disappeared, since it's already implicitly enabled via dependencies
* user wants to run a full test of Module_A and all of the code that it
depends upon (including Module_B)
* Module_B won't be tested and can't be explicitly enabled since it's hidden

Sure, there's a workaround where the user:
* disables Module_A
*re-configures
*enables Module_B
*re-configures
*enables Modules_A
*re-configures
but I don't think it's reasonable.

----

I sympathize with the original complaint, but I think that the CMake
configuration process is the wrong place to fix it. As I understand
(correct me if I'm missing something), the two big functions of testing in
ITK are:
* to be run manually by users, to provide confidence that a build is
functional
* to be automatically run on dashboards, reporting regressions

When a user runs tests manually, they probably want to test all of the
modules that are enabled (implicitly or explicitly). Testing only the
top-level explicitly-enabled modules does not inspire full confidence in
the software, as the underlying code (maybe something in Module_ITKCommon)
could be broken. If a user just wants to speed up their build times, they
should be disabling BUILD_TESTING all together. If an impatient developer
wants a fast build, as they're only testing a specific module(s), then
"make <modulename1>-all [<modulename2>-all]..." works just fine...as they
still have to type the "make" command either way.  In MSVC it's also easy
to select a specific target to build, instead of the solution or
"Build-all" target (or whatever it's called).

When tests are run on dashboards, I agree that it makes no sense to build
tests that will never be run, which easily happens if the configuration is
using the INCLUDE_LABEL option to only run tests with a certain label(s).
The solution is to fix the dashboard script. I think a simple solution is:
* add a new optional variable to itk_common.cmake: "CTEST_BUILD_ARGS"
* on itk_common.cmake:405, change "ctest_build()" to pass "${CTEST_BUILD_ARGS}"
as an argument
* a dashboard client script does: "set( CTEST_BUILD_ARGS TARGET ${
modulename}-all )"

The only issue is that "ctest_build()" can only take a single alternate
target as an argument, not a list of targets (improving this would require
some non-trivial changes to CMake). For the case of testing Remote modules,
however, only the one "${modulename}-all" target for the remote module
being tested needs to be built.










On Thu, Jan 16, 2014 at 5:31 PM, Matt McCormick
<matt.mccormick at kitware.com>wrote:

> Here is a patch that only build the tests for Modules specified
> explicitly when BUILD_DEFAULT_MODULES is OFF.
>
>   http://review.source.kitware.com/#/c/14063/
>
> Thanks,
> Matt
>
> On Thu, Jan 16, 2014 at 3:50 PM, Brian Helba <brian.helba at kitware.com>
> wrote:
> > For Makefile-based code generation, building with "make <modulename>-all"
> > should only build the module, tests, and its recursive code
> dependencies. It
> > should be possible to call
> > ctest_build(.. TARGET <modulename>-all ..) in the CTest driver script.
> >
> > I'm not sure about this, but I would hope that ctest_build can do the
> same
> > thing to directly build MSVC projects, as opposed to the entire solution.
> >
> >
> > On Thu, Jan 16, 2014 at 11:46 AM, Bradley Lowekamp <
> blowekamp at mail.nih.gov>
> > wrote:
> >>
> >>
> >> On Jan 16, 2014, at 11:28 AM, Bill Lorensen <bill.lorensen at gmail.com>
> >> wrote:
> >>
> >>
> >> It would be great if there were some way to turn off the building of
> >> the dependent module tests.
> >>
> >>
> >> +1
> >>
> >>
> >> This should be added to the infrastructure improvement list.
> >>
> >> I just tried to build the Core/Common module, and most of the toolkit
> was
> >> still turned on and the testing built.
> >>
> >> Brad
> >>
> >> _______________________________________________
> >> Powered by www.kitware.com
> >>
> >> Visit other Kitware open-source projects at
> >> http://www.kitware.com/opensource/opensource.html
> >>
> >> Kitware offers ITK Training Courses, for more information visit:
> >> http://kitware.com/products/protraining.php
> >>
> >> Please keep messages on-topic and check the ITK FAQ at:
> >> http://www.itk.org/Wiki/ITK_FAQ
> >>
> >> Follow this link to subscribe/unsubscribe:
> >> http://www.itk.org/mailman/listinfo/insight-developers
> >>
> >> _______________________________________________
> >> Community mailing list
> >> Community at itk.org
> >> http://public.kitware.com/cgi-bin/mailman/listinfo/community
> >>
> >
> >
> >
> > --
> > Brian Helba
> > Medical Imaging
> > Kitware, Inc.
> >
> > _______________________________________________
> > Powered by www.kitware.com
> >
> > Visit other Kitware open-source projects at
> > http://www.kitware.com/opensource/opensource.html
> >
> > Kitware offers ITK Training Courses, for more information visit:
> > http://kitware.com/products/protraining.php
> >
> > Please keep messages on-topic and check the ITK FAQ at:
> > http://www.itk.org/Wiki/ITK_FAQ
> >
> > Follow this link to subscribe/unsubscribe:
> > http://www.itk.org/mailman/listinfo/insight-developers
> >
> > _______________________________________________
> > Community mailing list
> > Community at itk.org
> > http://public.kitware.com/cgi-bin/mailman/listinfo/community
> >
>



-- 
Brian Helba
Medical Imaging
Kitware, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/community/attachments/20140116/c8d1409d/attachment-0002.html>
-------------- next part --------------
_______________________________________________
Powered by www.kitware.com

Visit other Kitware open-source projects at
http://www.kitware.com/opensource/opensource.html

Kitware offers ITK Training Courses, for more information visit:
http://kitware.com/products/protraining.php

Please keep messages on-topic and check the ITK FAQ at:
http://www.itk.org/Wiki/ITK_FAQ

Follow this link to subscribe/unsubscribe:
http://www.itk.org/mailman/listinfo/insight-developers