[Insight-developers] How to Validate the Registration Algorithms

[Luis Ibanez]

Hi Vaijanath


In practice there are two kinds of validation


A) The one that you make just to put in a paper
    and get a publication in your CV.


B) The one that you do for a real medical application
    where you care about what is going to happen to the
    patient.


-------------------------------------------------------

If you are dealing with (A), the typical approach is to
take two medical images, apply a known transformation to
one of them, and then use them as input to your
registration algorithm. In that context it is expected
that your registration will give back you a transform
that is very close to the one that was used for
"misregistering" the image in the first place.

The you do the same registration exercise with other
"popular" registration algorithms, and present a table
where you compare your error with the ones from other
algorithms. The error can be measured in many ways.
It depends a lot on the type of Transform that you
are using. In your case you seems to be using a
rigid transform in 2D, so you can probably use an
angle difference (between what you got and what you
expected as rotation), and a translation difference.

You could also compute the error by taking a number
of significant landmarks and computing the sum of
squared distances between their expected positions
and the actual positions when you resample with
the transform produced by your registration algorithm.

Getting other methods to compare against is now very
easy thanks to the wide range of registration methods
available in ITK. So you don't have to implement all
of them, just run experiments with your same images.


Such validation will produce a table like
==============================================================
Method             Error  Memory consumption   Computing time

MyGreatMethod       0.3      100Mb               10min
CompetitorMethodA   0.4       30Mb               20min
CompetitorMethodB   0.5       30Mb               60min
CompetitorMethodC   0.2      330Mb              120min
==============================================================

and of course you will conclude that your method does as well
as other competitor methods, or that it uses less memory or
that it runs faster.

If you are lucky, none of your reviewers will be the author
(or a big fan) of one of the competitor methods {A, B or C},
and you will get a paper accepted for publication.

If you are not lucky, you will get back reviews telling you
that you didn't cite the cardinal papers published earlier
by the reviewers, and comments indicating that you are trying
to solve a problem that has long ago been solved.



This validation (A) is *only useful for the vanity* purpose
of publishing papers and it is totally unscientific because
in reality a registration algorithm has a significant number
of parameters, and fine tunning those parameters is half of
the "engineering" of doing registration. Therefore, saying
that you compare with method "K" is totally useless unless
your statement is accompanied by the *full set of parameters*
that you used for the registration, and by the original images
that you used for your tests.  Furthermore, if you intend to
make claims regarding memory consumption and/or speed, you
must provide the source code that you used for the implementation
your and the other methods.  Without all this material is
*impossible* for anybody to replicate the results in your
table, and... a paper whose results cannot be replicated
it is not a scientific paper.

      "Reproducibility, not originality,
       is the signature of scientific work".


A skilled user of any of those registration algorithms will
certainly be capable of better exploiting the algorithm by
setting parameters better suited for the particular images
that you used as input. Your contribution will only be useful
if after your paper there is a dialog with skilled users of
the other methods and there is an iterative exchange where
they have the opportunity to fine-tune their methods to your
images, the same way you did for your own method.

---------

Now...
If you are in case (B), and you are doing this validation
because you *actually* plan to use your registration methods
for a real medical applications, such as diagnosis, surgery
planning, radiation treatment, or follow-up.  Then validation
can be done in a more honest way.

If you have a specific application for which your registration
is intended, then you have a target that tells you

      1) How much precision is needed for the final registration.
         E.g. 1mm ?,  10 microns ?

      2) Where in the images do you need more precision ?
         E.g. if you are dealing with brain images, you certainly
         care more about registering the brain than registering
         the nose. (unless the application is cosmetic surgery).

      3) How long is it acceptable to wait for your registration ?
         E.g. 1 hour ?, 10 seconds ?


In this context, things are easier to validate, (because they
actually mean something). You simply need to fine tune your method
to make sure that your results are withing the needed precision,
in the right places of the image, and that your method run under
the required maximum time.


---

   Please let us know if you have further questions,



      Thanks,



        Luis




------------------------------
Vaijanath Narasinga Rao wrote:

> Hi to all,
> Can someone tell me how do we validate the Registration algorithm. 
> Actually i have written an registration algorithm and since i am not 
> from Medical background do not really understand the validity of an 
> algorithm. The algorithm is quite simple, it finds the region and then 
> encloses it in an nearest rectangle. From here it gets the center of the 
> rectangle. this is done for both the images. From center coordinates one 
> can easily get the translation that needs to be applied. One you get the 
> desired translation, use the Co-relation index to find maximum 
> co-relation at an angle. Thus we get rotation.
> 
> The Application that I am making is an registration algorithm of 
> multi-modals.
> Your answers will help me in understanding things better,
> 
> Regards
> Vaijanath
> 
> _______________________________________________
> Insight-developers mailing list
> Insight-developers at itk.org
> http://www.itk.org/mailman/listinfo/insight-developers
> 
>