[Insight-developers] Change in ITK[master]: COMP: Intermittent test failure

[Bill Lorensen]

This failures look like a bug or parameter issue to me.

The large number of earlier failures were because the optimizer was
terminating on max iterations. My patch increased the max so that the
termination should be due to function tolerance and parameter difference
tolerance.

Perhaps the function tolerance needs to be reduced.

I think we'll have to let Ziv look at this.

Bill


On Fri, Sep 16, 2011 at 7:25 AM, Johnson, Hans J <hans-johnson at uiowa.edu>wrote:

>  Bill,
>
>  There were fewer failures, but there a still two similar failures:
>
>  Reason for stopping optimization:
> 	InitializationBiasedParticleSwarmOptimizer: successfuly converged after 95 iterations
> Known parameters   = [2, -2]   Estimated parameters = [2.1757, -2.09753]
> [Test 2 FAILURE]
>
>
> Hans
>
>
>
>   From: Bill Lorensen <bill.lorensen at gmail.com>
> Date: Thu, 15 Sep 2011 23:15:30 -0400
> To: "Yaniv, Ziv" <ZYaniv at childrensnational.org>
> Cc: Matt McCormick <matt.mccormick at kitware.com>, Hans Johnson <
> hans-johnson at uiowa.edu>, ITK <insight-developers at itk.org>
> Subject: Re: Change in ITK[master]: COMP: Intermittent test failure
>
>  Ziv,
>
> The problem with the test was that sometimes it was terminating on
> iterations. When this happened, the tolerance was not always satisfied.
>
> I upped the number of max iterations and I think the test should pass
> conosistently.
>
> We'll see.
>
> Usually for non-deterministic algorithms the test seeds the random number
> generator with a fixed number.
>
> On Thu, Sep 15, 2011 at 8:43 PM, Yaniv, Ziv <ZYaniv at childrensnational.org>wrote:
>
>> **
>>
>> Hi Bill,
>>
>> I was traveling and did not have internet connection for a day. Back
>> online. The PSO family of algorithms is stochastic and does not ensure
>> convergence to the local minimum. My initial implementation of the tests
>> dealt with this by running them multiple times and if a certain percentage
>> was successful then the test was declared successful. Perhaps this is not
>> the best way to test such algorithms but I do not have a better idea. The
>> nice feature of PSO is that it efficiently explores a rugged parameter
>> space. A common practice is to run the PSO and then initialize a standard
>> optimizer with the previous result, ensuring converges to the local minimum.
>>
>> What approach does ITK use to tests non-deterministic algorithms?
>>
>>               regards
>>                 Ziv
>>
>> --
>> Ziv Yaniv, PhD.,
>> Principal Investigator,
>> The Sheikh Zayed Institute for Pediatric Surgical Innovation
>> Children's National Medical Center
>> 6th Floor Main Hospital, Room M7740
>> 111 Michigan Ave., N.W
>> Washington, D.C 20010
>> Phone: 202-476-1288
>> email: zyaniv at childrensnational.org
>>
>>
>>
>>
>>
>> -----Original Message-----
>> From: Code Review [mailto:review at kitware.com <review at kitware.com>]
>> Sent: Thu 9/15/2011 6:25 PM
>> To: Bill Lorensen
>> Cc: Matt McCormick; Hans J. Johnson; Yaniv, Ziv
>> Subject: Change in ITK[master]: COMP: Intermittent test failure
>>
>> From Kitware Robot <kwrobot at kitware.com>:
>>
>> Kitware Robot has submitted this change and it was merged.
>>
>> Change subject: COMP: Intermittent test failure
>> ......................................................................
>>
>>
>> COMP: Intermittent test failure
>>
>> itkInitializationBiasedParticleSwarmOptimizerTest is failing
>> intermittenly. This patch enables more verbose (but not too verbose)
>> output to help track down the problem.
>>
>> It seems the test was failing a tolerance check on the results of the
>> optimization. This was due to hitting the max number of iterations
>> stop condition rather than the parameter tolerance stop
>> condition. This patch increases the max number of iterations, forcing
>> a parameter tolerance stop condition.
>>
>> It is not clear whether this is a deterministic algorithm. Should we
>> expect consistent answers for each run?
>>
>> This patch does not address the deterministic issue.
>>
>> Change-Id: Ie2110c51639f491cb7b8a6287b0eaeb1370c50bd
>> ---
>> M
>> Modules/Numerics/Optimizers/test/itkInitializationBiasedParticleSwarmOptimizerTest.cxx
>> 1 file changed, 45 insertions(+), 90 deletions(-)
>>
>> Approvals:
>>   Hans J. Johnson: Verified; Looks good to me, approved
>>   Bill Lorensen: Verified; Looks good to me, but someone else must approve
>>
>>
>> --
>> To view, visit http://review.source.kitware.com/2793
>> To unsubscribe, visit http://review.source.kitware.com/settings
>>
>> Gerrit-MessageType: merged
>> Gerrit-Change-Id: Ie2110c51639f491cb7b8a6287b0eaeb1370c50bd
>> Gerrit-PatchSet: 2
>> Gerrit-Project: ITK
>> Gerrit-Branch: master
>> Gerrit-Owner: Bill Lorensen <bill.lorensen at gmail.com>
>> Gerrit-Reviewer: Bill Lorensen <bill.lorensen at gmail.com>
>> Gerrit-Reviewer: Hans J. Johnson <hans-johnson at uiowa.edu>
>> Gerrit-Reviewer: Matt McCormick <matt.mccormick at kitware.com>
>> Gerrit-Reviewer: Ziv Yaniv <zyaniv at childrensnational.org>
>>
>>
>>
>
>
> ------------------------------
> Notice: This UI Health Care e-mail (including attachments) is covered by
> the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is
> confidential and may be legally privileged.  If you are not the intended
> recipient, you are hereby notified that any retention, dissemination,
> distribution, or copying of this communication is strictly prohibited.
> Please reply to the sender that you have received the message in error, then
> delete it.  Thank you.
> ------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/mailman/private/insight-developers/attachments/20110916/b84521a1/attachment-0001.htm>