[Insight-developers] Seeking advice about long running tests

[Matt McCormick]

Hi Brad,

> My ITK valgrind it taking too long. Just some test are just too long. How would I disable these tests on a dashboard script?

These tests could be disabled with

  set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)

> Also any idea what the change is coverage is?

Yes, the decrease is about 0.26 %.  Here are my local results:

Default run 1:
        Covered LOC:         123682
        Not covered LOC:     22557
        Total LOC:           146239
        Percentage Coverage: 84.58%

Default run 2:
        Covered LOC:         123671
        Not covered LOC:     22570
        Total LOC:           146241
        Percentage Coverage: 84.57%

-LE RUNS_LONG:
        Covered LOC:         123306
        Not covered LOC:     22942
        Total LOC:           146248
        Percentage Coverage: 84.31%

Thanks,
Matt


> On Sep 20, 2013, at 6:40 PM, Matt McCormick <matt.mccormick at kitware.com> wrote:
>
>> I did a visualization on test times [1] and it turns out that less
>> than 1% of the tests (0.97%) take of half (48.7%) of the test times!
>>
>> It would be good to add a "RUNS_LONG" label [2] to these tests so that
>> developers can run the test suite relatively quickly with
>>
>>  ctest -LE RUNS_LONG
>>
>> and the Visual Studio Debug Nightly builds can exclude them with
>>
>>  set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)
>>
>> These tests are mostly integration tests that test complicated
>> algorithms on semi-realistic data.  They have value, but leaving them
>> out in these situations should not do much harm relative to their time
>> overhead.
>>
>> Thanks,
>> Matt
>>
>> [1] http://review.source.kitware.com/#/c/12733/
>> [2] http://review.source.kitware.com/#/c/12734/
>>
>>
>> On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <bill.lorensen at gmail.com> wrote:
>>> In general, we do want to make sure the tests produce "correct" results and
>>> make sure there are no regressions. By having the Release build run a proper
>>> set of iterations, the developer can verify the correctness.
>>>
>>> That said, using a lower resolution input dataset could probably help most
>>> the these tests and the release/debug times would both be reduced.
>>>
>>>
>>>
>>> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <dlrdave at aol.com> wrote:
>>>>
>>>>
>>>>> I think it is OK to drop the iterations in the Debug tests. We are still
>>>>> exercising the code. If we
>>>>> need alternate baselines, we will still catch exceptions. This will also
>>>>> allow more valgrind tests
>>>>> to run to completion.
>>>>
>>>>
>>>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word “Fast” in
>>>> the test name... Perhaps we should at the very least consider a rename. 😊
>>>>
>>>> If it’s ok to drop the iterations in the Debug tests, then why not drop
>>>> the iterations in the Release tests, too? There is something to be said for
>>>> the “same” code being run for the test in both Debug and Release. If it’s
>>>> not the same, then I should definitely be able to tell just by glancing at
>>>> the source code for a test that there are Debug/non-Debug differences...
>>>>
>>>> I would say each individual test should be able to run in *seconds*, not
>>>> minutes or hours. Ideally, less than a second, but I realize that’s overly
>>>> ambitious with some ITK code.
>>>>
>>>> However, for the main purpose I have in mind here, namely reducing the
>>>> time burden on my volunteer machine, I would be absolutely *ecstatic* if all
>>>> of the 20 tests I listed in the original email were able to run in under 4
>>>> or 5 minutes each.
>>>>
>>>> Let me know if I can help out in some additional way (beyond just
>>>> continuing to submit the build to CDash...)
>>>>
>>>>
>>>> Thanks for the discussion, I appreciate it
>>>> D
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Unpaid intern in BillsBasement at noware dot com
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Kitware offers ITK Training Courses, for more information visit:
>>> http://kitware.com/products/protraining.php
>>>
>>> Please keep messages on-topic and check the ITK FAQ at:
>>> http://www.itk.org/Wiki/ITK_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>
>> _______________________________________________
>> Powered by www.kitware.com
>>
>> Visit other Kitware open-source projects at
>> http://www.kitware.com/opensource/opensource.html
>>
>> Kitware offers ITK Training Courses, for more information visit:
>> http://kitware.com/products/protraining.php
>>
>> Please keep messages on-topic and check the ITK FAQ at:
>> http://www.itk.org/Wiki/ITK_FAQ
>>
>> Follow this link to subscribe/unsubscribe:
>> http://www.itk.org/mailman/listinfo/insight-developers
>