[Insight-developers] Change in ITK[master]: COMP: Intermittent test failure

[Johnson, Hans J]

Agreed.

Sent from my iPhone

On Sep 16, 2011, at 7:24 AM, "Bill Lorensen" <bill.lorensen at gmail.com<mailto:bill.lorensen at gmail.com>> wrote:

This failures look like a bug or parameter issue to me.

The large number of earlier failures were because the optimizer was terminating on max iterations. My patch increased the max so that the termination should be due to function tolerance and parameter difference tolerance.

Perhaps the function tolerance needs to be reduced.

I think we'll have to let Ziv look at this.

Bill


On Fri, Sep 16, 2011 at 7:25 AM, Johnson, Hans J <<mailto:hans-johnson at uiowa.edu>hans-johnson at uiowa.edu<mailto:hans-johnson at uiowa.edu>> wrote:
Bill,

There were fewer failures, but there a still two similar failures:


Reason for stopping optimization:
        InitializationBiasedParticleSwarmOptimizer: successfuly converged after 95 iterations
Known parameters   = [2, -2]   Estimated parameters = [2.1757, -2.09753]
[Test 2 FAILURE]


Hans


From: Bill Lorensen <<mailto:bill.lorensen at gmail.com>bill.lorensen at gmail.com<mailto:bill.lorensen at gmail.com>>
Date: Thu, 15 Sep 2011 23:15:30 -0400
To: "Yaniv, Ziv" <<mailto:ZYaniv at childrensnational.org>ZYaniv at childrensnational.org<mailto:ZYaniv at childrensnational.org>>
Cc: Matt McCormick <<mailto:matt.mccormick at kitware.com>matt.mccormick at kitware.com<mailto:matt.mccormick at kitware.com>>, Hans Johnson <<mailto:hans-johnson at uiowa.edu>hans-johnson at uiowa.edu<mailto:hans-johnson at uiowa.edu>>, ITK <<mailto:insight-developers at itk.org>insight-developers at itk.org<mailto:insight-developers at itk.org>>
Subject: Re: Change in ITK[master]: COMP: Intermittent test failure

Ziv,

The problem with the test was that sometimes it was terminating on iterations. When this happened, the tolerance was not always satisfied.

I upped the number of max iterations and I think the test should pass conosistently.

We'll see.

Usually for non-deterministic algorithms the test seeds the random number generator with a fixed number.

On Thu, Sep 15, 2011 at 8:43 PM, Yaniv, Ziv <<mailto:ZYaniv at childrensnational.org>ZYaniv at childrensnational.org<mailto:ZYaniv at childrensnational.org>> wrote:

Hi Bill,

I was traveling and did not have internet connection for a day. Back online. The PSO family of algorithms is stochastic and does not ensure convergence to the local minimum. My initial implementation of the tests dealt with this by running them multiple times and if a certain percentage was successful then the test was declared successful. Perhaps this is not the best way to test such algorithms but I do not have a better idea. The nice feature of PSO is that it efficiently explores a rugged parameter space. A common practice is to run the PSO and then initialize a standard optimizer with the previous result, ensuring converges to the local minimum.

What approach does ITK use to tests non-deterministic algorithms?

              regards
                Ziv

--
Ziv Yaniv, PhD.,
Principal Investigator,
The Sheikh Zayed Institute for Pediatric Surgical Innovation
Children's National Medical Center
6th Floor Main Hospital, Room M7740
111 Michigan Ave., N.W
Washington, D.C 20010
Phone: 202-476-1288<tel:202-476-1288>
email: <mailto:zyaniv at childrensnational.org> zyaniv at childrensnational.org<mailto:zyaniv at childrensnational.org>





-----Original Message-----
From: Code Review [<mailto:review at kitware.com>mailto:review at kitware.com]
Sent: Thu 9/15/2011 6:25 PM
To: Bill Lorensen
Cc: Matt McCormick; Hans J. Johnson; Yaniv, Ziv
Subject: Change in ITK[master]: COMP: Intermittent test failure