[Insight-developers] Seeking advice about long running tests

[Bradley Lowekamp]

Matt,

This sounds like a good idea to me too.

My ITK valgrind it taking too long. Just some test are just too long. How would I disable these tests on a dashboard script?

Also any idea what the change is coverage is? 

Thanks,
Brad

On Sep 20, 2013, at 6:40 PM, Matt McCormick <matt.mccormick at kitware.com> wrote:

> I did a visualization on test times [1] and it turns out that less
> than 1% of the tests (0.97%) take of half (48.7%) of the test times!
> 
> It would be good to add a "RUNS_LONG" label [2] to these tests so that
> developers can run the test suite relatively quickly with
> 
>  ctest -LE RUNS_LONG
> 
> and the Visual Studio Debug Nightly builds can exclude them with
> 
>  set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)
> 
> These tests are mostly integration tests that test complicated
> algorithms on semi-realistic data.  They have value, but leaving them
> out in these situations should not do much harm relative to their time
> overhead.
> 
> Thanks,
> Matt
> 
> [1] http://review.source.kitware.com/#/c/12733/
> [2] http://review.source.kitware.com/#/c/12734/
> 
> 
> On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>> In general, we do want to make sure the tests produce "correct" results and
>> make sure there are no regressions. By having the Release build run a proper
>> set of iterations, the developer can verify the correctness.
>> 
>> That said, using a lower resolution input dataset could probably help most
>> the these tests and the release/debug times would both be reduced.
>> 
>> 
>> 
>> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <dlrdave at aol.com> wrote:
>>> 
>>> 
>>>> I think it is OK to drop the iterations in the Debug tests. We are still
>>>> exercising the code. If we
>>>> need alternate baselines, we will still catch exceptions. This will also
>>>> allow more valgrind tests
>>>> to run to completion.
>>> 
>>> 
>>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word “Fast” in
>>> the test name... Perhaps we should at the very least consider a rename. 😊
>>> 
>>> If it’s ok to drop the iterations in the Debug tests, then why not drop
>>> the iterations in the Release tests, too? There is something to be said for
>>> the “same” code being run for the test in both Debug and Release. If it’s
>>> not the same, then I should definitely be able to tell just by glancing at
>>> the source code for a test that there are Debug/non-Debug differences...
>>> 
>>> I would say each individual test should be able to run in *seconds*, not
>>> minutes or hours. Ideally, less than a second, but I realize that’s overly
>>> ambitious with some ITK code.
>>> 
>>> However, for the main purpose I have in mind here, namely reducing the
>>> time burden on my volunteer machine, I would be absolutely *ecstatic* if all
>>> of the 20 tests I listed in the original email were able to run in under 4
>>> or 5 minutes each.
>>> 
>>> Let me know if I can help out in some additional way (beyond just
>>> continuing to submit the build to CDash...)
>>> 
>>> 
>>> Thanks for the discussion, I appreciate it
>>> D
>>> 
>> 
>> 
>> 
>> 
>> --
>> Unpaid intern in BillsBasement at noware dot com
>> 
>> _______________________________________________
>> Powered by www.kitware.com
>> 
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>> 
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.php
>> 
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>> 
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>> 
> _______________________________________________
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.php
> 
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers