[Insight-developers] Very suspicious behavior.
Daniel Blezek
Blezek.Daniel at mayo.edu
Fri Feb 25 10:57:16 EST 2011
Hi Hans,
I can verify your test fails for different numbers of threads.
This machine has 8 cores. Fails at 7, 9, 12.
Will continue to look into this for you.
-dan
mi3c-pong:BRAINSFit-build blezek$ for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14
15 16;do echo "=============== Running with ${i} Threads ========= " ;
export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
BRAINSFitTest_RigidRotGeomNoMasks$ ; done
=============== Running with 1 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 2 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 3 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 4 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 5 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 6 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 7 Threads =========
0% tests passed, 1 tests failed out of 1
=============== Running with 8 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 9 Threads =========
0% tests passed, 1 tests failed out of 1
The following tests FAILED:
28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
Errors while running CTest
=============== Running with 10 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 11 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 12 Threads =========
0% tests passed, 1 tests failed out of 1
The following tests FAILED:
28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
Errors while running CTest
=============== Running with 13 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 14 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 15 Threads =========
100% tests passed, 0 tests failed out of 1
=============== Running with 16 Threads =========
100% tests passed, 0 tests failed out of 1
On 2/24/11 8:29 PM, "Johnson, Hans J" <hans-johnson at uiowa.edu> wrote:
> Hello ITK Experts:
>
> I have a really nasty problem that is almost certainly some sort algorithmic
> anomaly that occurs during the creation of multiple threads under the right
> data conditions. The failure is a lot more than just numerical precision, it
> is about 15 degrees and several mm off.
>
> Mark Scully has been chasing this down for several days now, and we have been
> able to narrow down the environments for which this occurs.
>
> The problem attempting to be solved is a straight forward
> Versor3DTransformOptimizer, MattesMutualInformation, LiniearInterpolator ITK
> registration process between two T1 weighted images.
>
> ---To cause failure run the process with exactly 4 threads (it also fails with
> 7, 14,15 threads, but not with 1,2,3,5,6,8,9,10,11,12,13, or 16 threads).
>
> ---We have replicated the problem on Window 32bi, Linux 32/64, and Mac 64 bit
> OS. Additionally, on mac we've tested in both Debug and Release mode, and
> we've built it against both ITKv3 and ITKv4. The results are consistent in
> that they consistently only fail when the thread count is one of {4,7,14,15} ,
> and passing with any of the other listed successful number of threads (see
> listing at end of message).
>
> ---Changing the metric from MattesMutualInformation to MeanSquareError allows
> this to pass with any number of threads.
>
> --ITK is built with OptimizedRegistration in ITKv3 (In ITKv4
> OptimizedRegistration is the default).
>
> Create a bash script with the following contents and run to see the behavior.
> The first instance of the test will pass, the second will fail.
>
> \/\//\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\
> /\/\/\/\/
> #!/bin/sh
> svn checkout --username slicerbot --password slicer
> https://www.nitrc.org/svn/brains/StandAloneApps/StandAloneBRAINSFit
> StandAloneBRAINSFit
> mkdir StandAloneBRAINSFit-build
> cd StandAloneBRAINSFit-build
> cmake ../StandAloneBRAINSFit
> make
> cd BRAINSFit-build
>
> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=1
> ctest -R RigidRotGeom
>
> echo export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=4
> ctest -R RigidRotGeom
> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>
>
> Any hints on a strategy to help identify and correct the failure would be
> GREATLY appreciated.
>
> Thanks,
> Hans
>
>
> [hjohnson at hjhomebuildbox BRAINSFit-build]$ for i in 1 2 3 4 5 6 7 8 9 10 11 12
> 13 14 15 16;do echo "=============== Running with ${i} Threads ========= " ;
> export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${i} ; ctest -R
> BRAINSFitTest_RigidRotGeomNoMasks$ ; done
> =============== Running with 1 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 184.79 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 184.81 sec
> =============== Running with 2 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 140.10 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 140.13 sec
> =============== Running with 3 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 54.98 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 55.01 sec
> =============== Running with 4 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 22.80 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 22.83 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 5 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 35.23 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 35.29 sec
> =============== Running with 6 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 29.61 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 29.63 sec
> =============== Running with 7 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 55.38 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 55.41 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 8 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.25 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 17.26 sec
> =============== Running with 9 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 27.49 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 27.51 sec
> =============== Running with 10 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 36.89 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 36.92 sec
> =============== Running with 11 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 19.71 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 19.72 sec
> =============== Running with 12 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 17.97 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 17.98 sec
> =============== Running with 13 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 34.96 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 34.98 sec
> =============== Running with 14 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 18.18 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 18.20 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 15 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ...***Failed 17.45 sec
>
> 0% tests passed, 1 tests failed out of 1
>
> Total Test time (real) = 17.48 sec
>
> The following tests FAILED:
> 28 - BRAINSFitTest_RigidRotGeomNoMasks (Failed)
> Errors while running CTest
> =============== Running with 16 Threads =========
> Test project
> /Users/hjohnson/src/StandAloneApps/BF-ITKv4-Release/BRAINSFit-build
> Start 28: BRAINSFitTest_RigidRotGeomNoMasks
> 1/1 Test #28: BRAINSFitTest_RigidRotGeomNoMasks ... Passed 23.14 sec
>
> 100% tests passed, 0 tests failed out of 1
>
> Total Test time (real) = 23.15 sec
>
>
>
>
> Notice: This UI Health Care e-mail (including attachments) is covered by the
> Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential
> and may be legally privileged. If you are not the intended recipient, you are
> hereby notified that any retention, dissemination, distribution, or copying of
> this communication is strictly prohibited. Please reply to the sender that
> you have received the message in error, then delete it. Thank you.
>
>
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
--
Daniel Blezek, PhD
Medical Imaging Informatics Innovation Center
P 127 or (77) 8 8886
T 507 538 8886
E blezek.daniel at mayo.edu
Mayo Clinic
200 First St. S.W.
Harwick SL-44
Rochester, MN 55905
mayoclinic.org
"It is more complicated than you think." -- RFC 1925
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/mailman/private/insight-developers/attachments/20110225/c6a02482/attachment.htm>
More information about the Insight-developers
mailing list