[Insight-developers] Test framework

Blezek, Daniel J. Blezek.Daniel at mayo.edu
Tue Dec 30 10:21:29 EST 2008 

Bill,

  If all we want to do is make testing easier, then a set of macros modeled after gtest or boost would be great.  No need to drag along yet another code base.  Still, it might be good to kick the tires.  Google tends to produce well thought out code.  You know that I tend to adapt code from experts rather than develop things on my own, so I'm biased here.  I'm more inclined to write a test using gtest because I find it easier than ITK's environment.  But I'm also lazy...

Cheers,
-dan

-----Original Message-----
From: insight-developers-bounces at itk.org [mailto:insight-developers-bounces at itk.org] On Behalf Of Bill Lorensen
Sent: Tuesday, December 30, 2008 9:11 AM
To: Blezek, Daniel J.
Cc: insight-developers at itk.org
Subject: Re: [Insight-developers] Test framework

Dan,

I agree with all of your points. And, we could still run the google tests from our test executables. This would allow us to support the existing image regression tests.

However, why should we add a another set of code we will need to support when all we really need is to add a few macros to help our testing.

Bill

On Tue, Dec 30, 2008 at 9:51 AM, Blezek, Daniel J.
<Blezek.Daniel at mayo.edu> wrote:
> Hi Bill,
>
>  I wasn't clear on some points.  While my examples used MD5, there is no reason comparisons to expected images couldn't do the same.  The test would just become an ASSERT.  Though all our tests reside in one executable that can be called from the command line, I still hook them into ctest/cdash.  Each test is run individually, just like the existing ITK framework.  The main executable returns a non zero status if any test fails.  Basically, I've just replaced the executables that are built for ctest to run.  I find Google Test easy and efficient to use.
>
>  I would like to point out that what we have now is _exactly_ like using the class in an application.  In reality VTK and ITK tests have always been more of examples than true regression tests.  The post by Steve way down at the end started this discussion by pointing out that ITK's example-tests are much more verbose than they need to be, precisely because they are example-oriented.
>
>  My lab's use of Google Test/ctest/cdash is driven by testing alone, not for example code.  This is because we are not building a public toolkit, but internal.  Thus my examples were very short, and perhaps cryptic.  Each is designed to exercise the code and provide coverage (25% and climbing).
>
>  In reality, the ITK and Google Test could mix and mingle very easily.  January is very, very busy for me, OTW I'd write a short Insight Journal article on Google Test applied to ITK.
>
> Merry Christmas to all, and Happy New Year, -dan
>
>
>
> -----Original Message-----
> From: Bill Lorensen [mailto:bill.lorensen at gmail.com]
> Sent: Tuesday, December 30, 2008 7:41 AM
> To: Gaëtan Lehmann
> Cc: Blezek, Daniel J.; Daniel Blezek; insight-developers at itk.org
> Subject: Re: [Insight-developers] Test framework
>
> I have looked a bit more at both proposed test frameworks. I'm not sure how helpful they will be for itk testing.
> For example, the google testing has, as Dan pointed out, several ctest like features. Well, we don't need ctest features in another test framework since we already have ctest.
>
> Also, google test can run many tests from one executable. We also do that. Because of the way we do cdash reporting, I think we would still run the tests one at a time.
>
> I think we would still have to run the new tests from our current test executables to take advantage of ctest control and cdash reporting.
> This would still permit regression testing and thread control.
>
> Finally, for many of our classes, writing a test is very similar to using the class in an application. This is a good exercise for the developer and illustrates how the filter can be used. I think the google tests appear a bit more cryptic.
>
> Perhaps we should look at the two proposed frameworks and provide some of their ideas into our testing framework.
>
> Bill
>
> On Tue, Dec 30, 2008 at 5:33 AM, Gaëtan Lehmann <gaetan.lehmann at jouy.inra.fr> wrote:
>>
>> Le 29 déc. 08 à 22:12, Blezek, Daniel J. a écrit :
>>
>>> Yep, it is cool.  We've been using it extensively and I really like it.
>>> On reflection, I'm not sure it will be a drop in replacement for the
>>> test routines currently in use, without some messing around with the
>>> internals.  The framework manages test discovery and execution and
>>> leaves it intentionally opaque.  However, a good strategy might be to
>>> leave the existing tests in place and start using a new test
>>> framework, i.e., make itkCommonTests3 use the new framework.
>>>
>>
>> Hi,
>>
>> I don't think we should fully switch to a new test framework, even if
>> it's limited to the new tests.
>> The current framework has some good points that no other test
>> framework would have currently, like the expected/produced images
>> comparison in cdash for example.
>>
>> My understanding is that Bill was looking for a unit test framework to
>> complete the current test suite with unit tests, wich is nice IMO, not
>> to fully replace the current one. Right?
>>
>> I actually have some tests which would benefit of such framework -
>> like
>> http://voxel.jouy.inra.fr/darcs/contrib-itk/binaryAttributeMorphology/
>> attribute_values_test.cxx, one of the worth I've ever wrote (works in
>> Debug mode only and, according to the result on the IJ, doesn't pass
>> on other computers than mine).
>>
>> Gaëtan
>>
>>
>>> I documented our experiences with Google Test on the wiki as you
>>> suggested:
>>> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Google
>>> _Te
>>> st
>>>
>>> Cheers,
>>> -dan
>>>
>>> -----Original Message-----
>>> From: insight-developers-bounces at itk.org
>>> [mailto:insight-developers-bounces at itk.org] On Behalf Of Bill
>>> Lorensen
>>> Sent: Monday, December 29, 2008 12:17 PM
>>> To: Daniel Blezek
>>> Cc: insight-developers at itk.org
>>> Subject: Re: [Insight-developers] Test framework
>>>
>>> Dan,
>>>
>>> Sounds cool. Can you add some info to:
>>>
>>> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage
>>>
>>> Bill
>>>
>>> On Mon, Dec 29, 2008 at 1:14 PM, Daniel Blezek
>>> <daniel.blezek at gmail.com>
>>> wrote:
>>>>
>>>> Hi Bill,
>>>>
>>>> I would add the Google Test framework to the mix.  We've been using
>>>> it extensively in our lab.  It meets 2, 3, 4, 5, 6, 7.  Not sure
>>>> about
>>>
>>>> platform support across _all_ ITK's platform, but I've never had a
>>>> compile warning from it nor any memory leaks.  Given that they
>>>> support
>>>
>>>> several embedded platforms, I'd bet it compiles cleanly on all our
>>>> platforms.  It's a very minimalist library, mostly implemented
>>>> through
>>>
>>>> macros and supports many of the nice ctest features, i.e. run every
>>>> N'th test, run all matching tests, etc.
>>>>
>>>> http://code.google.com/p/googletest/
>>>>
>>>> I found a CMake macro that parses the source code and automatically
>>>> adds tests making it trivial to add new tests:
>>>>
>>>> macro(ADD_GOOGLE_TESTS executable)
>>>> foreach ( source ${ARGN} )
>>>>  file(READ "${source}" contents)
>>>>  string(REGEX MATCHALL "TEST_?F?\\(([A-Za-z_0-9 ,]+)\\)"
>>>> found_tests
>>>> ${contents})
>>>>  foreach(hit ${found_tests})
>>>>    string(REGEX REPLACE ".*\\(([A-Za-z_0-9]+)[,
>>>
>>> ]*([A-Za-z_0-9]+)\\).*"
>>>>
>>>> "\\1.\\2" test_name ${hit})
>>>>    add_test(${test_name} ${executable} --gtest_filter=${test_name}
>>>> ${MI3CTestingDir})
>>>>  endforeach(hit)
>>>> endforeach()
>>>> endmacro()
>>>>
>>>>
>>>> # Add all tests found in the source code, calling the executable to
>>>> run them add_google_tests ( ${EXECUTABLE_OUTPUT_PATH}/NoOp
>>>> ${mi3cTestSource})
>>>>
>>>> The main looks like this:
>>>>
>>>> #include <gtest/gtest.h>
>>>>
>>>> int main(int argc, char* argv[])
>>>> {
>>>> testing::InitGoogleTest ( &argc, argv ); return RUN_ALL_TESTS(); }
>>>>
>>>> and a test looks like this (we use MD5 hashes to test for pass/fail):
>>>>
>>>> #include <iostream>
>>>>
>>>> #include <mi3cImage.h>
>>>> #include <mi3cImageLoader.h>
>>>> #include <gtest/gtest.h>
>>>> #include <NoOp.h>
>>>>
>>>> TEST(IO, LoadCT) {
>>>> mi3c::ImageLoader loader;
>>>> mi3c::Image::Pointer image = loader.setFilename ( dataFinder.getFile
>>>
>>>> ( "CT.hdr" ) ).execute();
>>>> ASSERT_EQ ( "c1d43aaa5b991431a9daa1dc4b55dbb1", image->getMD5() ); }
>>>>
>>>>
>>>> Fixtures are also supported to remove redundant code.
>>>>
>>>> Cheers,
>>>> -dan
>>>>
>>>> On Mon, Dec 29, 2008 at 10:35 AM, Bill Lorensen
>>>> <bill.lorensen at gmail.com>
>>>> wrote:
>>>>>
>>>>> I agree that we should look at a unit test facility.
>>>>>
>>>>> We should start a list of requirements. For example,
>>>>>
>>>>> 1) Must support all itk platforms.
>>>>> 2) We must be able to distribute it with itk.
>>>>> 3) It must fit within our test harness facility. Recall that we try
>>>>> to minimize the number of executables by combining large numbers of
>>>>> tests into FooTests.cxx files.
>>>>> 4) It must be compatible with cmake/ctest/cdash. For example, a
>>>>> test must be able to "return EXIT_SUCCESS" and "return EXIT_FAILURE".
>>>>> 5) It should not add complexity to an already complex testing
>>>
>>> process.
>>>>>
>>>>> 6) It must be compatible with itk's strict attention to compile
>>>>> warnings and dynamic memory anlysis. In other words, it must not
>>>>> produce warnings or purify defects.
>>>>> 7) It should have a minimal source footprint.
>>>>>
>>>>> These are just a few requirements off the top of my head. I'll add
>>>>> them to the new page when I get a chance,
>>>>>
>>>>> Bill
>>>>>
>>>>> On Mon, Dec 29, 2008 at 10:47 AM, Luis Ibanez
>>>>> <luis.ibanez at kitware.com>
>>>>> wrote:
>>>>>>
>>>>>> Hi Mathieu,
>>>>>>
>>>>>> Minor correction:
>>>>>>
>>>>>> The text of the UnitTestCpp license:
>>>>>>
>>>>>> https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp/Unit
>>>>>> T
>>>>>> est++/COPYING
>>>>>>
>>>>>> doesn't correspond to a BSD license:
>>>>>> http://www.opensource.org/licenses/bsd-license.php
>>>>>>
>>>>>> but to an MIT license:
>>>>>> http://www.opensource.org/licenses/mit-license.php
>>>>>>
>>>>>> which is a technicality after all, since both licenses could be
>>>
>>> combined
>>>>>>
>>>>>> without any conflict.   :-)
>>>>>>
>>>>>>
>>>>>> -----
>>>>>>
>>>>>>
>>>>>> Could you tell us more about your experience with UnitTestCpp  ?
>>>>>>
>>>>>> or if you prefer, could you add your comments to the Wiki page:
>>>>>>
>>>>>> http://www.itk.org/Wiki/Proposals:Increasing_ITK_Code_Coverage#Uni
>>>>>> t
>>>>>> TestCpp
>>>>>>
>>>>>>
>>>>>> The size of the package looks fine. It could fit neatly in
>>>>>> Insight/Utilities...
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>>  Luis
>>>>>>
>>>>>>
>>>>>> --------------------------
>>>>>> Mathieu Malaterre wrote:
>>>>>>>
>>>>>>> On Mon, Dec 29, 2008 at 5:29 AM, Steve M. Robbins
>>>>>>> <steve at sumost.ca>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> On Wed, Dec 10, 2008 at 03:51:56PM -0500, Bill Lorensen wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> Given the success of "Adopt a Bug", maybe we should initiate a
>>>>>>>>> "Cover a Class".
>>>>>>>>
>>>>>>>> Now that I have some holiday time, I'd like to raise the issue
>>>>>>>> of
>>>
>>>>>>>> unit testing frameworks.
>>>>>>>>
>>>>>>>> The few test codes that I've looked at are all written out
>>>>>>>> long-hand, without benefit of a modern unit testing framework
>>>>>>>> such as CppUnit or Boost.Test.
>>>>>>>
>>>>>>>
>>>>>>> ... or UnitTestCpp :)
>>>>>>>
>>>>>>> http://unittest-cpp.sourceforge.net/
>>>>>>>
>>>>>>> Which comes with a BSD like license ;)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> https://unittest-cpp.svn.sourceforge.net/svnroot/unittest-cpp/Uni
>>>>>>> t
>>>>>>> Test++/COPYING
>>>>>>>
>>>>>>> I wrote a cmakelists.txt file for it:
>>>>>>>
>>>>>>> http://gdcm.svn.sf.net/viewvc/gdcm/Sandbox/UnitTest%2B%2B/
>>>>>>>
>>>>>>> src files are very lightweight "du -sk src" return 712
>>>>>>>
>>>>>>>
>>>>>>> 2cts
>>>>>>
>>>>> _______________________________________________
>>>>> Insight-developers mailing list
>>>>> Insight-developers at itk.org
>>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>>
>>>>
>>> _______________________________________________
>>> Insight-developers mailing list
>>> Insight-developers at itk.org
>>> http://www.itk.org/mailman/listinfo/insight-developers
>>> _______________________________________________
>>> Insight-developers mailing list
>>> Insight-developers at itk.org
>>> http://www.itk.org/mailman/listinfo/insight-developers
>>
>> --
>> Gaëtan Lehmann
>> Biologie du Développement et de la Reproduction INRA de Jouy-en-Josas
>> (France)
>> tel: +33 1 34 65 29 66    fax: 01 34 65 29 09
>> http://voxel.jouy.inra.fr  http://www.mandriva.org http://www.itk.org
>> http://www.clavier-dvorak.org
>>
>>
>
_______________________________________________
Insight-developers mailing list
Insight-developers at itk.org
http://www.itk.org/mailman/listinfo/insight-developers

More information about the Insight-developers mailing list