[Insight-developers] [ITK Community] Testing of Remote Modules

[Matt McCormick]

Hi Brian,

If all the tests or all the dependencies (Module_B, etc) of a module
want to be tested, then BUILD_DEFAULT_MODULES should be ON.  It is
reasonable to expect that turning on one module will only create tests
for that module.  Otherwise, in practice with the current tree,
testing dependency propagation will literally cause half of the
modules to be built no matter how the build is started..

Thanks,
Matt

On Thu, Jan 16, 2014 at 7:55 PM, Brian Helba <brian.helba at kitware.com> wrote:
> This patch breaks the following use case:
> * Module_A depends upon Module_B
> * user disables BUILD_DEFAULT_MODULES, enables Module_A
> * note that after a re-configure, the CMake option for Module_B has
> disappeared, since it's already implicitly enabled via dependencies
> * user wants to run a full test of Module_A and all of the code that it
> depends upon (including Module_B)
> * Module_B won't be tested and can't be explicitly enabled since it's hidden
>
> Sure, there's a workaround where the user:
> * disables Module_A
> *re-configures
> *enables Module_B
> *re-configures
> *enables Modules_A
> *re-configures
> but I don't think it's reasonable.
>
> ----
>
> I sympathize with the original complaint, but I think that the CMake
> configuration process is the wrong place to fix it. As I understand (correct
> me if I'm missing something), the two big functions of testing in ITK are:
> * to be run manually by users, to provide confidence that a build is
> functional
> * to be automatically run on dashboards, reporting regressions
>
> When a user runs tests manually, they probably want to test all of the
> modules that are enabled (implicitly or explicitly). Testing only the
> top-level explicitly-enabled modules does not inspire full confidence in the
> software, as the underlying code (maybe something in Module_ITKCommon) could
> be broken. If a user just wants to speed up their build times, they should
> be disabling BUILD_TESTING all together. If an impatient developer wants a
> fast build, as they're only testing a specific module(s), then "make
> <modulename1>-all [<modulename2>-all]..." works just fine...as they still
> have to type the "make" command either way.  In MSVC it's also easy to
> select a specific target to build, instead of the solution or "Build-all"
> target (or whatever it's called).
>
> When tests are run on dashboards, I agree that it makes no sense to build
> tests that will never be run, which easily happens if the configuration is
> using the INCLUDE_LABEL option to only run tests with a certain label(s).
> The solution is to fix the dashboard script. I think a simple solution is:
> * add a new optional variable to itk_common.cmake: "CTEST_BUILD_ARGS"
> * on itk_common.cmake:405, change "ctest_build()" to pass
> "${CTEST_BUILD_ARGS}" as an argument
> * a dashboard client script does: "set( CTEST_BUILD_ARGS TARGET
> ${modulename}-all )"
>
> The only issue is that "ctest_build()" can only take a single alternate
> target as an argument, not a list of targets (improving this would require
> some non-trivial changes to CMake). For the case of testing Remote modules,
> however, only the one "${modulename}-all" target for the remote module being
> tested needs to be built.
>
>
>
>
>
>
>
>
>
>
> On Thu, Jan 16, 2014 at 5:31 PM, Matt McCormick <matt.mccormick at kitware.com>
> wrote:
>>
>> Here is a patch that only build the tests for Modules specified
>> explicitly when BUILD_DEFAULT_MODULES is OFF.
>>
>>   http://review.source.kitware.com/#/c/14063/
>>
>> Thanks,
>> Matt
>>
>> On Thu, Jan 16, 2014 at 3:50 PM, Brian Helba <brian.helba at kitware.com>
>> wrote:
>> > For Makefile-based code generation, building with "make
>> > <modulename>-all"
>> > should only build the module, tests, and its recursive code
>> > dependencies. It
>> > should be possible to call
>> > ctest_build(.. TARGET <modulename>-all ..) in the CTest driver script.
>> >
>> > I'm not sure about this, but I would hope that ctest_build can do the
>> > same
>> > thing to directly build MSVC projects, as opposed to the entire
>> > solution.
>> >
>> >
>> > On Thu, Jan 16, 2014 at 11:46 AM, Bradley Lowekamp
>> > <blowekamp at mail.nih.gov>
>> > wrote:
>> >>
>> >>
>> >> On Jan 16, 2014, at 11:28 AM, Bill Lorensen <bill.lorensen at gmail.com>
>> >> wrote:
>> >>
>> >>
>> >> It would be great if there were some way to turn off the building of
>> >> the dependent module tests.
>> >>
>> >>
>> >> +1
>> >>
>> >>
>> >> This should be added to the infrastructure improvement list.
>> >>
>> >> I just tried to build the Core/Common module, and most of the toolkit
>> >> was
>> >> still turned on and the testing built.
>> >>
>> >> Brad
>> >>
>> >> _______________________________________________
>> >> Powered by www.kitware.com
>> >>
>> >> Visit other Kitware open-source projects at
>> >> http://www.kitware.com/opensource/opensource.html
>> >>
>> >> Kitware offers ITK Training Courses, for more information visit:
>> >> http://kitware.com/products/protraining.php
>> >>
>> >> Please keep messages on-topic and check the ITK FAQ at:
>> >> http://www.itk.org/Wiki/ITK_FAQ
>> >>
>> >> Follow this link to subscribe/unsubscribe:
>> >> http://www.itk.org/mailman/listinfo/insight-developers
>> >>
>> >> _______________________________________________
>> >> Community mailing list
>> >> Community at itk.org
>> >> http://public.kitware.com/cgi-bin/mailman/listinfo/community
>> >>
>> >
>> >
>> >
>> > --
>> > Brian Helba
>> > Medical Imaging
>> > Kitware, Inc.
>> >
>> > _______________________________________________
>> > Powered by www.kitware.com
>> >
>> > Visit other Kitware open-source projects at
>> > http://www.kitware.com/opensource/opensource.html
>> >
>> > Kitware offers ITK Training Courses, for more information visit:
>> > http://kitware.com/products/protraining.php
>> >
>> > Please keep messages on-topic and check the ITK FAQ at:
>> > http://www.itk.org/Wiki/ITK_FAQ
>> >
>> > Follow this link to subscribe/unsubscribe:
>> > http://www.itk.org/mailman/listinfo/insight-developers
>> >
>> > _______________________________________________
>> > Community mailing list
>> > Community at itk.org
>> > http://public.kitware.com/cgi-bin/mailman/listinfo/community
>> >
>
>
>
>
> --
> Brian Helba
> Medical Imaging
> Kitware, Inc.