[Insight-developers] Seeking advice about long running tests

[Matt McCormick]

I did a visualization on test times [1] and it turns out that less
than 1% of the tests (0.97%) take of half (48.7%) of the test times!

It would be good to add a "RUNS_LONG" label [2] to these tests so that
developers can run the test suite relatively quickly with

  ctest -LE RUNS_LONG

and the Visual Studio Debug Nightly builds can exclude them with

  set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)

These tests are mostly integration tests that test complicated
algorithms on semi-realistic data.  They have value, but leaving them
out in these situations should not do much harm relative to their time
overhead.

Thanks,
Matt

[1] http://review.source.kitware.com/#/c/12733/
[2] http://review.source.kitware.com/#/c/12734/


On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
> In general, we do want to make sure the tests produce "correct" results and
> make sure there are no regressions. By having the Release build run a proper
> set of iterations, the developer can verify the correctness.
>
> That said, using a lower resolution input dataset could probably help most
> the these tests and the release/debug times would both be reduced.
>
>
>
> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <dlrdave at aol.com> wrote:
>>
>>
>> > I think it is OK to drop the iterations in the Debug tests. We are still
>> > exercising the code. If we
>> > need alternate baselines, we will still catch exceptions. This will also
>> > allow more valgrind tests
>> > to run to completion.
>>
>>
>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word “Fast” in
>> the test name... Perhaps we should at the very least consider a rename. 😊
>>
>> If it’s ok to drop the iterations in the Debug tests, then why not drop
>> the iterations in the Release tests, too? There is something to be said for
>> the “same” code being run for the test in both Debug and Release. If it’s
>> not the same, then I should definitely be able to tell just by glancing at
>> the source code for a test that there are Debug/non-Debug differences...
>>
>> I would say each individual test should be able to run in *seconds*, not
>> minutes or hours. Ideally, less than a second, but I realize that’s overly
>> ambitious with some ITK code.
>>
>> However, for the main purpose I have in mind here, namely reducing the
>> time burden on my volunteer machine, I would be absolutely *ecstatic* if all
>> of the 20 tests I listed in the original email were able to run in under 4
>> or 5 minutes each.
>>
>> Let me know if I can help out in some additional way (beyond just
>> continuing to submit the build to CDash...)
>>
>>
>> Thanks for the discussion, I appreciate it
>> D
>>
>
>
>
>
> --
> Unpaid intern in BillsBasement at noware dot com
>
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.php
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
>