[Insight-developers] Test framework
Gaëtan Lehmann
gaetan.lehmann at jouy.inra.fr
Tue Dec 30 05:33:24 EST 2008
Le 29 déc. 08 à 22:12, Blezek, Daniel J. a écrit :
> Yep, it is cool. We've been using it extensively and I really like
> it.
> On reflection, I'm not sure it will be a drop in replacement for the
> test routines currently in use, without some messing around with the
> internals. The framework manages test discovery and execution and
> leaves it intentionally opaque. However, a good strategy might be to
> leave the existing tests in place and start using a new test
> framework,
> i.e., make itkCommonTests3 use the new framework.
>
Hi,
I don't think we should fully switch to a new test framework, even if
it's limited to the new tests.
The current framework has some good points that no other test
framework would have currently, like the expected/produced images
comparison in cdash for example.
My understanding is that Bill was looking for a unit test framework to
complete the current test suite with unit tests, wich is nice IMO, not
to fully replace the current one. Right?
I actually have some tests which would benefit of such framework -
like http://voxel.jouy.inra.fr/darcs/contrib-itk/binaryAttributeMorphology/attribute_values_test.cxx
, one of the worth I've ever wrote (works in Debug mode only and,
according to the result on the IJ, doesn't pass on other computers
than mine).
Gaëtan
> I documented our experiences with Google Test on the wiki as you
> suggested:
> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Google_Te
> st
>
> Cheers,
> -dan
>
> -----Original Message-----
> From: insight-developers-bounces at itk.org
> [mailto:insight-developers-bounces at itk.org] On Behalf Of Bill Lorensen
> Sent: Monday, December 29, 2008 12:17 PM
> To: Daniel Blezek
> Cc: insight-developers at itk.org
> Subject: Re: [Insight-developers] Test framework
>
> Dan,
>
> Sounds cool. Can you add some info to:
>
> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage
>
> Bill
>
> On Mon, Dec 29, 2008 at 1:14 PM, Daniel Blezek <daniel.blezek at gmail.com
> >
> wrote:
>> Hi Bill,
>>
>> I would add the Google Test framework to the mix. We've been using
>> it extensively in our lab. It meets 2, 3, 4, 5, 6, 7. Not sure
>> about
>
>> platform support across _all_ ITK's platform, but I've never had a
>> compile warning from it nor any memory leaks. Given that they
>> support
>
>> several embedded platforms, I'd bet it compiles cleanly on all our
>> platforms. It's a very minimalist library, mostly implemented
>> through
>
>> macros and supports many of the nice ctest features, i.e. run every
>> N'th test, run all matching tests, etc.
>>
>> http://code.google.com/p/googletest/
>>
>> I found a CMake macro that parses the source code and automatically
>> adds tests making it trivial to add new tests:
>>
>> macro(ADD_GOOGLE_TESTS executable)
>> foreach ( source ${ARGN} )
>> file(READ "${source}" contents)
>> string(REGEX MATCHALL "TEST_?F?\\(([A-Za-z_0-9 ,]+)\\)"
>> found_tests
>> ${contents})
>> foreach(hit ${found_tests})
>> string(REGEX REPLACE ".*\\(([A-Za-z_0-9]+)[,
> ]*([A-Za-z_0-9]+)\\).*"
>> "\\1.\\2" test_name ${hit})
>> add_test(${test_name} ${executable} --gtest_filter=${test_name}
>> ${MI3CTestingDir})
>> endforeach(hit)
>> endforeach()
>> endmacro()
>>
>>
>> # Add all tests found in the source code, calling the executable to
>> run them add_google_tests ( ${EXECUTABLE_OUTPUT_PATH}/NoOp
>> ${mi3cTestSource})
>>
>> The main looks like this:
>>
>> #include <gtest/gtest.h>
>>
>> int main(int argc, char* argv[])
>> {
>> testing::InitGoogleTest ( &argc, argv );
>> return RUN_ALL_TESTS();
>> }
>>
>> and a test looks like this (we use MD5 hashes to test for pass/fail):
>>
>> #include <iostream>
>>
>> #include <mi3cImage.h>
>> #include <mi3cImageLoader.h>
>> #include <gtest/gtest.h>
>> #include <NoOp.h>
>>
>> TEST(IO, LoadCT) {
>> mi3c::ImageLoader loader;
>> mi3c::Image::Pointer image = loader.setFilename ( dataFinder.getFile
>
>> ( "CT.hdr" ) ).execute();
>> ASSERT_EQ ( "c1d43aaa5b991431a9daa1dc4b55dbb1", image->getMD5() ); }
>>
>>
>> Fixtures are also supported to remove redundant code.
>>
>> Cheers,
>> -dan
>>
>> On Mon, Dec 29, 2008 at 10:35 AM, Bill Lorensen
>> <bill.lorensen at gmail.com>
>> wrote:
>>>
>>> I agree that we should look at a unit test facility.
>>>
>>> We should start a list of requirements. For example,
>>>
>>> 1) Must support all itk platforms.
>>> 2) We must be able to distribute it with itk.
>>> 3) It must fit within our test harness facility. Recall that we try
>>> to minimize the number of executables by combining large numbers of
>>> tests into FooTests.cxx files.
>>> 4) It must be compatible with cmake/ctest/cdash. For example, a test
>>> must be able to "return EXIT_SUCCESS" and "return EXIT_FAILURE".
>>> 5) It should not add complexity to an already complex testing
> process.
>>> 6) It must be compatible with itk's strict attention to compile
>>> warnings and dynamic memory anlysis. In other words, it must not
>>> produce warnings or purify defects.
>>> 7) It should have a minimal source footprint.
>>>
>>> These are just a few requirements off the top of my head. I'll add
>>> them to the new page when I get a chance,
>>>
>>> Bill
>>>
>>> On Mon, Dec 29, 2008 at 10:47 AM, Luis Ibanez
>>> <luis.ibanez at kitware.com>
>>> wrote:
>>>>
>>>> Hi Mathieu,
>>>>
>>>> Minor correction:
>>>>
>>>> The text of the UnitTestCpp license:
>>>>
>>>> https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp/UnitT
>>>> est++/COPYING
>>>>
>>>> doesn't correspond to a BSD license:
>>>> http://www.opensource.org/licenses/bsd-license.php
>>>>
>>>> but to an MIT license:
>>>> http://www.opensource.org/licenses/mit-license.php
>>>>
>>>> which is a technicality after all, since both licenses could be
> combined
>>>> without any conflict. :-)
>>>>
>>>>
>>>> -----
>>>>
>>>>
>>>> Could you tell us more about your experience with UnitTestCpp ?
>>>>
>>>> or if you prefer, could you add your comments to the Wiki page:
>>>>
>>>> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Unit
>>>> TestCpp
>>>>
>>>>
>>>> The size of the package looks fine. It could fit neatly in
>>>> Insight/Utilities...
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>> Luis
>>>>
>>>>
>>>> --------------------------
>>>> Mathieu Malaterre wrote:
>>>>>
>>>>> On Mon, Dec 29, 2008 at 5:29 AM, Steve M. Robbins
>>>>> <steve at sumost.ca>
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> On Wed, Dec 10, 2008 at 03:51:56PM -0500, Bill Lorensen wrote:
>>>>>>
>>>>>>
>>>>>>> Given the success of "Adopt a Bug", maybe we should initiate a
>>>>>>> "Cover a Class".
>>>>>>
>>>>>> Now that I have some holiday time, I'd like to raise the issue of
>
>>>>>> unit testing frameworks.
>>>>>>
>>>>>> The few test codes that I've looked at are all written out
>>>>>> long-hand, without benefit of a modern unit testing framework
>>>>>> such as CppUnit or Boost.Test.
>>>>>
>>>>>
>>>>> ... or UnitTestCpp :)
>>>>>
>>>>> http://unittest-cpp.sourceforge.net/
>>>>>
>>>>> Which comes with a BSD like license ;)
>>>>>
>>>>>
>>>>>
>>>>> https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp/Unit
>>>>> Test++/COPYING
>>>>>
>>>>> I wrote a cmakelists.txt file for it:
>>>>>
>>>>> http://gdcm.svn.sf.net/viewvc/gdcm/Sandbox/UnitTest%2B%2B/
>>>>>
>>>>> src files are very lightweight "du -sk src" return 712
>>>>>
>>>>>
>>>>> 2cts
>>>>
>>> _______________________________________________
>>> Insight-developers mailing list
>>> Insight-developers at itk.org
>>> http://www.itk.org/mailman/listinfo/insight-developers
>>
>>
> _______________________________________________
> Insight-developers mailing list
> Insight-developers at itk.org
> http://www.itk.org/mailman/listinfo/insight-developers
> _______________________________________________
> Insight-developers mailing list
> Insight-developers at itk.org
> http://www.itk.org/mailman/listinfo/insight-developers
--
Gaëtan Lehmann
Biologie du Développement et de la Reproduction
INRA de Jouy-en-Josas (France)
tel: +33 1 34 65 29 66 fax: 01 34 65 29 09
http://voxel.jouy.inra.fr http://www.mandriva.org
http://www.itk.org http://www.clavier-dvorak.org
More information about the Insight-developers
mailing list