[Insight-users] Rigid registration testing and reproducibility

Wed Jan 28 12:47:17 EST 2009

On Wed, Jan 28, 2009 at 12:17 PM, Andriy Fedorov <fedorov at bwh.harvard.edu>wrote:

> Daniel, Bill -- thanks for replies.
>
> I wonder, how are these multiple baselines generated? Are they
> collected from running the test on different platforms?

yes.. floating precision implementations, ceil /floor/rnd implementations,
compiler optimizations etc cause minor differences in the result, even when
running the same algorithm, initialized with the same seed. We just check
multiple baselines if they are deemend acceptable.

Or by
> adjusting rotation and translation components in all possible
> combinations by some (0.1?) tolerance, and applying the transform to
> the input? Seems like a daunting task in 3d...

No

>
>
> From what I see, ImageRegistration13Test doesn't use baselines, it
> compares the true transform parameters not to exceed thresholds --
> 0.025 for scale and 1.0 for translations.

But as Dan mentioned its hard for most transforms. The right way to go about
it is generate the target images manually with a known transform. Then after
registration, you'd multiply the obtained transform with the known
transform's inverse and check the L2 norm / bending energy or its equivalent
against a tolerance.

Given the work it requires, its simpler to just compute the RMSE over
source/target images.

> On Wed, Jan 28, 2009 at 11:30 AM, Bill Lorensen <bill.lorensen at gmail.com>
> wrote:
> > Also, their can be multiple baselines for a given test. If you look at:
> >
> >
> http://public.kitware.com/cgi-bin/viewcvs.cgi/Testing/Data/Baseline/Registration/?root=Insight
> >
> > you will see that some tests have multiple baselines because of the
> > type of variability discussed in this thread.
> >
> > For example, ImageRegistration13Test has 5 baselines.
> >
> > Bill
> >
> > On Wed, Jan 28, 2009 at 11:14 AM, Blezek, Daniel J.
> > <Blezek.Daniel at mayo.edu> wrote:
> >> Just 2 cents here.
> >>
> >> 1) For a linear registration, you could expect the registration results
> >> to be ~0.1 degrees and ~0.1 mm cross platform.  The major source of this
> >> problem is differences in floating point representations under different
> >> compilers/hardware.  I shouldn't worry about this, it'd be difficult for
> >> a human to see.  For something like BSplines, you might have a bit more
> >> error.
> >>
> >> 2) A very simple way to do this is to do a forward transformation using
> >> one transform (ground truth), then the inverse transform of the newly
> >> calculated transform.  I think this is called Target Registration Error
> >> by Fitzpatrick and Co., but you should look it up.  This is where you
> >> need to decide on your tolerance.  As Karthik mentioned, only simple ITK
> >> transforms have an inverse, which is really a shame.  They are so
> >> useful, even if they are numeric.
> >>
> >> 2a) I suppose you could forward transform the same point using two
> >> different transforms and see how far apart they are.  This seems
> >> reasonable, but you'd have to sample a bunch of points to account for
> >> rotation, and transform centers, etc.  And you'd only get a distance
> >> measure, not a rotational measure.
> >>
> >> For transforms with an inverse, you can do what you are asking, and it
> >> would be a valuable contribution to ITK, but it's not general, as not
> >> all transforms support an inverse.  And you could always test the
> >> transform you care about...
> >>
> >> Incoherent as usual,
> >> -dan
> >>
> >> -----Original Message-----
> >> From: insight-users-bounces at itk.org
> >> [mailto:insight-users-bounces at itk.org] On Behalf Of Andriy Fedorov
> >> Sent: Tuesday, January 27, 2009 6:18 PM
> >> To: ITK Users
> >> Cc: Miller, James V (GE, Research)
> >> Subject: [Insight-users] Rigid registration testing and reproducibility
> >>
> >> Hi,
> >>
> >> I would like to do regression tests of rigid registration with real
> >> images, and compare the result with the baseline transform. Here are two
> >> related questions.
> >>
> >> 1) ITK registration experts, could you speculate on what is the normal
> >> variability in the rigid registration results run with the same
> >> parameters, metric initialized with the same seed, when run across
> >> different platforms? What could be the sources of this variability,
> >> given the same initialization and same parameters?
> >>
> >> 2) If I understand correctly, the current testing of registration that
> >> comes with ITK generates a synthetic image with known transform
> >> parameters, which are compared with the registration-derived transforms.
> >>
> >> The testing framework implemented in itkTestMain.h allows to compare
> >> pixelwise two images, but this does not seem to be practical for
> >> registration regression testing, because of the variability in the
> >> registration results I mentioned earlier. Accounting for this
> >> variability using tolerance values of just 1 pixel hugely increases the
> >> test runtime, but in my experience, the comparison may still fail.
> >>
> >> Would it not make sense to augment itkTestMain.h with the capability to
> >> compare not only images, but transforms? Is this a valid feature
> >> request, or am I missing something?
> >>
> >> Thanks
> >>
> >> Andriy Fedorov
> >> _______________________________________________
> >> Insight-users mailing list
> >> Insight-users at itk.org
> >> http://www.itk.org/mailman/listinfo/insight-users
> >> _______________________________________________
> >> Insight-users mailing list
> >> Insight-users at itk.org
> >> http://www.itk.org/mailman/listinfo/insight-users
> >>
> >
> _______________________________________________
> Insight-users mailing list
> Insight-users at itk.org
> http://www.itk.org/mailman/listinfo/insight-users
>

-- 
Karthik Krishnan
R&D Engineer,
Kitware Inc.
Ph: 518 371 3971 x119
Fax: 518 371 3971
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/pipermail/insight-users/attachments/20090128/0694eac3/attachment.htm>