[Insight-developers] Seeking advice about long running tests

[David Cole]

Hi there.


First of all: kudos to all of you on dashboard progress, and fixing issues with Windows Debug builds. I have now submitted my first ever entirely green dashboard row using the Microsoft compiler in a Debug build for ITK!


However...... 😊


I recently started submitting ITK dashboards from “mug.neocisinc” and observe the following today:


- 20 longest running ITK tests (today):
itkFastMarchingImageFilterTest_wm_multipleSeeds_StrictTopo 2h 37m 54s 70ms
itkFastMarchingImageFilterTest_wm_multipleSeeds_NoHandlesTopo 2h 37m 28s 440ms
itkFastMarchingImageFilterTest_wm_multipleSeeds_NoTopo 2h 27m 34s 760ms
itkBSplineExponentialImageRegistrationTest 22m 9s 470ms
itkExponentialImageRegistrationTest 19m 55s 840ms
itkFastMarchingImageFilterTest_torus_multipleSeeds_NoHandlesTopo 13m 37s 950ms
itkFastMarchingImageFilterTest_torus_multipleSeeds_StrictTopo 13m 34s 700ms
itkSimpleImageRegistrationTest 12m 16s 690ms
itkSimpleImageRegistrationTest2 11m 32s 560ms
itkFastMarchingImageFilterTest_torus_multipleSeeds_NoTopo 11m 4s 70ms
itkTimeVaryingVelocityFieldImageRegistrationTest 8m 12s 240ms
SpatialObjectToImage2 7m 4s 710ms
ImageRegistration15Test 5m 6s 410ms
ImageRegistration14Test 3m 56s
itkParameterizationQuadEdgeMeshFilterTestb0_c3_s0 3m 47s 950ms
itkParameterizationQuadEdgeMeshFilterTestb1_c3_s0 3m 47s 150ms
DeformableRegistration13Test1 3m 3s 480ms
itkTimeVaryingBSplineVelocityFieldImageRegistrationTest 2m 54s 310ms
itkSyNImageRegistrationTest 2m 38s 180ms
DeformableRegistration12Test1 2m 27s 350ms




(from this Debug build using the Microsoft Visual Studio 2012 compiler: http://open.cdash.org/buildSummary.php?buildid=2985530 )


Does anybody in this community have any advice about how I can get the Debug dashboard test suite to run in a more reasonable amount of time? (Other than simply excluding these 20 tests or so by name?)


It seems completely ridiculous to me that I have to set a test timeout value of greater than 10 minutes (let alone 2.5 HOURS) for an individual test just for the 10 longest running ITK tests here...... especially when the ENTIRE TEST SUITE finishes in less than 6 minutes for the Release build.


Can these tests be pared down to use smaller data, or can they be marked as requiring seriously long timeout values using the test TIMEOUT property, or can they be optionally added by people willing to test such long running tests, or perhaps optionally avoided by people like me who are reluctant to run such long running tests .... ?? There’s got to be something we can do.



Thanks,

David C.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/pipermail/insight-developers/attachments/20130802/4f035635/attachment.htm>