[Insight-developers] Re: Problems with new registrationoptimizations

[Stephen R. Aylward]

This touches upon a developing topic: ITK consistency when run single versus
multi-threaded.

At this time, we have been able to relatively easily implement algorithms so
that they produce the same results (i.e., have the same implementation
logic) when run single or multi-threaded.  This stability holds for the new
multi-threaded registration framework.  Recall that this consistency
constraint has such small tolerance that it, for example, requires the use
of a specific random seed with our implementation of a specific random
number generator for certain tests to pass.

ISSUE: I am confident that we could continue to ensure the same consistency
(e.g., make sure the same random number sequence is generated when run
single or multi-threaded) in all future work, but the complexity introduced
if we must always maintain that extreme consistency is perhaps detrimental
to multi-threaded algorithm efficiency and ease of understanding.

PROPOSED SOLUTION: I propose that in the future we ALLOW the tolerance
needed for a test to pass to be adjusted, if that adjustment is needed to
account for non-significant variations due to running the method single vs.
multi-threaded.    The additional uncertainty/inconsistency in the results
should not allow for variations in the interpretation of the functionality
of the algorithm and should not allow for code errors to go undetected.  The
additional uncertainty/inconsistency should allow, for examples, for
non-significant changes in results, where those changes are similar in
magnitude to what would be expected if a different random seed was used in a
test.

Agreed?

Stephen

=======================================
Stephen R. Aylward, Ph.D.
Chief Medical Scientist
Kitware, Inc. - Chapel Hill Office
http://www.kitware.com
Phone: (518)371-3971 x300
 

> -----Original Message-----
> From: 
> insight-developers-bounces+stephen.aylward=kitware.com at itk.org
>  
> [mailto:insight-developers-bounces+stephen.aylward=kitware.com
> @itk.org] On Behalf Of Matt Turek
> Sent: Monday, December 03, 2007 12:16 PM
> To: Bill Lorensen
> Cc: ITK
> Subject: [Insight-developers] Re: Problems with new 
> registrationoptimizations
> 
> 
> Bill,
> 
> I'll take a look at the failing tests. We've been focused on 
> cleaning up the itk optimized dashboard.
> 
> Matt
> 
> 
> Bill Lorensen wrote:
> > Matt,
> >  
> > Looks like a crash in one of the metrics on those platforms 
> that have 
> > turned on the new optimizations:
> > 
> http://www.itk.org/Testing/Sites/zion.kitware/Linux-g++-4.1/20071201-0
> > 
> 100-Nightly/Results/__Testing_Code_Algorithms_itkMattesMutualInformati
> > onImageToImageMetricTest.html 
> > 
> <http://www.itk.org/Testing/Sites/zion.kitware/Linux-g++-4.1/20071201-
> > 
> 0100-Nightly/Results/__Testing_Code_Algorithms_itkMattesMutualInformat
> > ionImageToImageMetricTest.html>
> >  
> > Also, 13 registration regression tests are failing on the optimized 
> > registraion enabled gcc build:
> > 
> http://www.itk.org/Testing/Sites/zion.kitware/Linux-g++-4.1/20071201-0
> > 100-Nightly/Test.html
> >  
> > Should we expect the exact results?
> >  
> > Bill
> >  
> 
> _______________________________________________
> Insight-developers mailing list
> Insight-developers at itk.org
> http://www.itk.org/mailman/listinfo/insight-developers