[Insight-users] Rigid registration testing and reproducibility

[Andriy Fedorov]

On Wed, Jan 28, 2009 at 2:29 PM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> Baselines are generated on each platform when they are deemed to be
> correct, but not exact as a current baseline.
>
> Regression testing is meant to detect changes in code  and/or
> platforms.

This last item is exactly my objective.

But how exactly can one possibly generate baseline images from
different platforms, if those platforms are not under one's control
and dynamically change (eg, in a real dashboard with experimental
builds)?

I don't see how the currently implemented mechanisms of testing can
help to resolve this. If I am missing something, I am ready to provide
detailed test cases of what I am trying to achieve and learn, since I
failed to find similar testing examples in either ITK or Slicer so far
(regression testing of rigid registration with baselines).

My current solution is to implement custom testing within the
registration module, and compare transforms, as Jim, Karthik and Dan
previously suggested.


>
> Bill
>
>
> On Wed, Jan 28, 2009 at 12:17 PM, Andriy Fedorov
> <fedorov at bwh.harvard.edu> wrote:
>> Daniel, Bill -- thanks for replies.
>>
>> I wonder, how are these multiple baselines generated? Are they
>> collected from running the test on different platforms? Or by
>> adjusting rotation and translation components in all possible
>> combinations by some (0.1?) tolerance, and applying the transform to
>> the input? Seems like a daunting task in 3d...
>>
>> From what I see, ImageRegistration13Test doesn't use baselines, it
>> compares the true transform parameters not to exceed thresholds --
>> 0.025 for scale and 1.0 for translations.
>>
>>
>> On Wed, Jan 28, 2009 at 11:30 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>> Also, their can be multiple baselines for a given test. If you look at:
>>>
>>> http://public.kitware.com/cgi-bin/viewcvs.cgi/Testing/Data/Baseline/Registration/?root=Insight
>>>
>>> you will see that some tests have multiple baselines because of the
>>> type of variability discussed in this thread.
>>>
>>> For example, ImageRegistration13Test has 5 baselines.
>>>
>>> Bill
>>>
>>> On Wed, Jan 28, 2009 at 11:14 AM, Blezek, Daniel J.
>>> <Blezek.Daniel at mayo.edu> wrote:
>>>> Just 2 cents here.
>>>>
>>>> 1) For a linear registration, you could expect the registration results
>>>> to be ~0.1 degrees and ~0.1 mm cross platform.  The major source of this
>>>> problem is differences in floating point representations under different
>>>> compilers/hardware.  I shouldn't worry about this, it'd be difficult for
>>>> a human to see.  For something like BSplines, you might have a bit more
>>>> error.
>>>>
>>>> 2) A very simple way to do this is to do a forward transformation using
>>>> one transform (ground truth), then the inverse transform of the newly
>>>> calculated transform.  I think this is called Target Registration Error
>>>> by Fitzpatrick and Co., but you should look it up.  This is where you
>>>> need to decide on your tolerance.  As Karthik mentioned, only simple ITK
>>>> transforms have an inverse, which is really a shame.  They are so
>>>> useful, even if they are numeric.
>>>>
>>>> 2a) I suppose you could forward transform the same point using two
>>>> different transforms and see how far apart they are.  This seems
>>>> reasonable, but you'd have to sample a bunch of points to account for
>>>> rotation, and transform centers, etc.  And you'd only get a distance
>>>> measure, not a rotational measure.
>>>>
>>>> For transforms with an inverse, you can do what you are asking, and it
>>>> would be a valuable contribution to ITK, but it's not general, as not
>>>> all transforms support an inverse.  And you could always test the
>>>> transform you care about...
>>>>
>>>> Incoherent as usual,
>>>> -dan
>>>>
>>>> -----Original Message-----
>>>> From: insight-users-bounces at itk.org
>>>> [mailto:insight-users-bounces at itk.org] On Behalf Of Andriy Fedorov
>>>> Sent: Tuesday, January 27, 2009 6:18 PM
>>>> To: ITK Users
>>>> Cc: Miller, James V (GE, Research)
>>>> Subject: [Insight-users] Rigid registration testing and reproducibility
>>>>
>>>> Hi,
>>>>
>>>> I would like to do regression tests of rigid registration with real
>>>> images, and compare the result with the baseline transform. Here are two
>>>> related questions.
>>>>
>>>> 1) ITK registration experts, could you speculate on what is the normal
>>>> variability in the rigid registration results run with the same
>>>> parameters, metric initialized with the same seed, when run across
>>>> different platforms? What could be the sources of this variability,
>>>> given the same initialization and same parameters?
>>>>
>>>> 2) If I understand correctly, the current testing of registration that
>>>> comes with ITK generates a synthetic image with known transform
>>>> parameters, which are compared with the registration-derived transforms.
>>>>
>>>> The testing framework implemented in itkTestMain.h allows to compare
>>>> pixelwise two images, but this does not seem to be practical for
>>>> registration regression testing, because of the variability in the
>>>> registration results I mentioned earlier. Accounting for this
>>>> variability using tolerance values of just 1 pixel hugely increases the
>>>> test runtime, but in my experience, the comparison may still fail.
>>>>
>>>> Would it not make sense to augment itkTestMain.h with the capability to
>>>> compare not only images, but transforms? Is this a valid feature
>>>> request, or am I missing something?
>>>>
>>>> Thanks
>>>>
>>>> Andriy Fedorov
>>>> _______________________________________________
>>>> Insight-users mailing list
>>>> Insight-users at itk.org
>>>> http://www.itk.org/mailman/listinfo/insight-users
>>>> _______________________________________________
>>>> Insight-users mailing list
>>>> Insight-users at itk.org
>>>> http://www.itk.org/mailman/listinfo/insight-users
>>>>
>>>
>>
>