[Insight-developers] Optimizer termination criteria

[Zachary Pincus]

Hello folks,

I've been chatting with the folks who contributed the SPSA optimizer to 
ITK (on the users list), partially about the termination criteria for 
that class. Initially we thought that adding a few different criteria 
would be good, but on reflection (a) there are simply too many 
reasonable options for termination criteria to support all of them in 
one class, (b) this would contribute even more to the balkanization of 
the optimizer API that had been previously mentioned, and (c) this 
wouldn't do any good for any other optimizers.

So the suggestion that we came up with was that in general optimizers 
might be best to not implement any fancy termination criteria,* and 
instead rely on user-provided callbacks to terminate optimization. 
Then, provided with a prefab set of callbacks for most reasonable types 
of termination criteria (size of parameter change, size of objective 
function value change, size of gradient), the user could simply plug in 
the appropriate criteria for their task.

So: Is this a good idea? Or is there enough of a performance penalty 
from adding callbacks that requiring a callback for every optimizer 
would be infeasible? Or some other objection?

If this is a worthy goal, I think it would be possible to refactor 
existing optimizers so that their "termination criteria API" stays the 
same, while they, behind the scenes, just use callbacks. This way, 
backwards compatibility could be maintained, while still allowing users 
to remove these criteria and add their own.

What do you guys think? Does this suggestion even approach a reasonable 
work-to-reward tradeoff, compared to some of the other stuff on the ITK 
roadmap?

Thanks,

Zach Pincus

Department of Biochemistry and Program in Biomedical Informatics
Stanford University School of Medicine


* Or some optimizer-specific criteria that is best evaluated in the 
optimizer itself, e.g. the OnePlusOne optimizer's 
frobenius-norm-of-the-A-matrix criteria.