[Insight-users] Optimizer spikes

[Luis Ibanez]

Hi Steve,

This have been discussed recently in the list,
please take a look at the archives of May 2006.

The reason for having spikes in the metric is that
the optimizer may jump over the minimum value of
the metric and then bounce back. The bouncing back
doesn't guarrantee that is going to hit the cost
function in a place where the value is lower.

This is because the direction of the jump is
computed using the local derivatives, but the
lenght of the jump is given by the learning rate
(in the case of this optimizer).

If you see this behavior, it probably indicates
that you are using a learning rate that is too
large for the current landscape in which the
optimizer is walking.

Note that there is no reason to assume that the
learning rate that is good during the initial stage
of an optimization, will also be good for the final
stage. You may want to consider adding an observer
to the optimizer, that will regulate the value of
the learning rate after the optimization has progressed
for several iterations.


    Regards,


       Luis



===================
Steven Boyd wrote:
> Hello,
> 
> When observing the progress of rigid registration using the 
> QuaternionRigidTransformGradientDescentOptimizer via the metric value 
> (i.e., MattesMutualInformation) spikes occur, and sometimes last for 2-3 
> iterations.
> 
> Can somebody explain why there are spikes in the metric?  I would it's 
> related to a change in step size of the optimizer, but I'm not sure why 
> this would result in the spikes.
> 
> Regards,
> 
> Steve
> 
>