[Insight-developers] Re: Problems with new registrationoptimizations

[Bill Lorensen]

Stephen,

In principle I agree, but we still have to know when a change to the code
produces different results. The current case is a good one. There are bugs
in the current optimized registration code that seem to have been discovered
because our regression testing found a minor variation in the baseline
images.

You mention complexity. Currently, only the regression tests need to specify
a specific seed for the random numbers. If a user doesn't care about exact
reproduction of results from release to release, then they need not specify
the seed.

Bill


On Dec 3, 2007 1:01 PM, Stephen R. Aylward <Stephen.Aylward at kitware.com>
wrote:

> This touches upon a developing topic: ITK consistency when run single
> versus
> multi-threaded.
>
> At this time, we have been able to relatively easily implement algorithms
> so
> that they produce the same results (i.e., have the same implementation
> logic) when run single or multi-threaded.  This stability holds for the
> new
> multi-threaded registration framework.  Recall that this consistency
> constraint has such small tolerance that it, for example, requires the use
> of a specific random seed with our implementation of a specific random
> number generator for certain tests to pass.
>
> ISSUE: I am confident that we could continue to ensure the same
> consistency
> (e.g., make sure the same random number sequence is generated when run
> single or multi-threaded) in all future work, but the complexity
> introduced
> if we must always maintain that extreme consistency is perhaps detrimental
> to multi-threaded algorithm efficiency and ease of understanding.
>
> PROPOSED SOLUTION: I propose that in the future we ALLOW the tolerance
> needed for a test to pass to be adjusted, if that adjustment is needed to
> account for non-significant variations due to running the method single
> vs.
> multi-threaded.    The additional uncertainty/inconsistency in the results
> should not allow for variations in the interpretation of the functionality
> of the algorithm and should not allow for code errors to go undetected.
>  The
> additional uncertainty/inconsistency should allow, for examples, for
> non-significant changes in results, where those changes are similar in
> magnitude to what would be expected if a different random seed was used in
> a
> test.
>
> Agreed?
>
> Stephen
>
> =======================================
> Stephen R. Aylward, Ph.D.
> Chief Medical Scientist
> Kitware, Inc. - Chapel Hill Office
> http://www.kitware.com
> Phone: (518)371-3971 x300
>
>
> > -----Original Message-----
> > From:
> > insight-developers-bounces+stephen.aylward=kitware.com at itk.org
> >
> > [mailto:insight-developers-bounces+stephen.aylward=kitware.com
> > @itk.org] On Behalf Of Matt Turek
> > Sent: Monday, December 03, 2007 12:16 PM
> > To: Bill Lorensen
> > Cc: ITK
> > Subject: [Insight-developers] Re: Problems with new
> > registrationoptimizations
> >
> >
> > Bill,
> >
> > I'll take a look at the failing tests. We've been focused on
> > cleaning up the itk optimized dashboard.
> >
> > Matt
> >
> >
> > Bill Lorensen wrote:
> > > Matt,
> > >
> > > Looks like a crash in one of the metrics on those platforms
> > that have
> > > turned on the new optimizations:
> > >
> > http://www.itk.org/Testing/Sites/zion.kitware/Linux-g++-4.1/20071201-0
> > >
> > 100-Nightly/Results/__Testing_Code_Algorithms_itkMattesMutualInformati
> > > onImageToImageMetricTest.html
> > >
> > <http://www.itk.org/Testing/Sites/zion.kitware/Linux-g++-4.1/20071201-
> > >
> > 0100-Nightly/Results/__Testing_Code_Algorithms_itkMattesMutualInformat
> > > ionImageToImageMetricTest.html>
> > >
> > > Also, 13 registration regression tests are failing on the optimized
> > > registraion enabled gcc build:
> > >
> > http://www.itk.org/Testing/Sites/zion.kitware/Linux-g++-4.1/20071201-0
> > > 100-Nightly/Test.html
> > >
> > > Should we expect the exact results?
> > >
> > > Bill
> > >
> >
> > _______________________________________________
> > Insight-developers mailing list
> > Insight-developers at itk.org
> > http://www.itk.org/mailman/listinfo/insight-developers
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.itk.org/mailman/private/insight-developers/attachments/20071203/788e8453/attachment.htm