[Insight-developers] Seeking advice about long running tests

[Bill Lorensen]

+1


On Sat, Sep 21, 2013 at 8:48 AM, Johnson, Hans J <hans-johnson at uiowa.edu> wrote:
> Matt,
>
> I agree with this approach!
>
> Hans
>
> On 9/20/13 5:40 PM, "Matt McCormick" <matt.mccormick at kitware.com> wrote:
>
>>I did a visualization on test times [1] and it turns out that less
>>than 1% of the tests (0.97%) take of half (48.7%) of the test times!
>>
>>It would be good to add a "RUNS_LONG" label [2] to these tests so that
>>developers can run the test suite relatively quickly with
>>
>>  ctest -LE RUNS_LONG
>>
>>and the Visual Studio Debug Nightly builds can exclude them with
>>
>>  set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)
>>
>>These tests are mostly integration tests that test complicated
>>algorithms on semi-realistic data.  They have value, but leaving them
>>out in these situations should not do much harm relative to their time
>>overhead.
>>
>>Thanks,
>>Matt
>>
>>[1] http://review.source.kitware.com/#/c/12733/
>>[2] http://review.source.kitware.com/#/c/12734/
>>
>>
>>On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <bill.lorensen at gmail.com>
>>wrote:
>>> In general, we do want to make sure the tests produce "correct" results
>>>and
>>> make sure there are no regressions. By having the Release build run a
>>>proper
>>> set of iterations, the developer can verify the correctness.
>>>
>>> That said, using a lower resolution input dataset could probably help
>>>most
>>> the these tests and the release/debug times would both be reduced.
>>>
>>>
>>>
>>> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <dlrdave at aol.com> wrote:
>>>>
>>>>
>>>> > I think it is OK to drop the iterations in the Debug tests. We are
>>>>still
>>>> > exercising the code. If we
>>>> > need alternate baselines, we will still catch exceptions. This will
>>>>also
>>>> > allow more valgrind tests
>>>> > to run to completion.
>>>>
>>>>
>>>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word “Fast”
>>>>in
>>>> the test name... Perhaps we should at the very least consider a
>>>>rename. 😊
>>>>
>>>> If it’s ok to drop the iterations in the Debug tests, then why not drop
>>>> the iterations in the Release tests, too? There is something to be
>>>>said for
>>>> the “same” code being run for the test in both Debug and Release. If
>>>>it’s
>>>> not the same, then I should definitely be able to tell just by
>>>>glancing at
>>>> the source code for a test that there are Debug/non-Debug
>>>>differences...
>>>>
>>>> I would say each individual test should be able to run in *seconds*,
>>>>not
>>>> minutes or hours. Ideally, less than a second, but I realize that’s
>>>>overly
>>>> ambitious with some ITK code.
>>>>
>>>> However, for the main purpose I have in mind here, namely reducing the
>>>> time burden on my volunteer machine, I would be absolutely *ecstatic*
>>>>if all
>>>> of the 20 tests I listed in the original email were able to run in
>>>>under 4
>>>> or 5 minutes each.
>>>>
>>>> Let me know if I can help out in some additional way (beyond just
>>>> continuing to submit the build to CDash...)
>>>>
>>>>
>>>> Thanks for the discussion, I appreciate it
>>>> D
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> Unpaid intern in BillsBasement at noware dot com
>>>
>>> _______________________________________________
>>> Powered by www.kitware.com
>>>
>>> Visit other Kitware open-source projects at
>>> http://www.kitware.com/opensource/opensource.html
>>>
>>> Kitware offers ITK Training Courses, for more information visit:
>>> http://kitware.com/products/protraining.php
>>>
>>> Please keep messages on-topic and check the ITK FAQ at:
>>> http://www.itk.org/Wiki/ITK_FAQ
>>>
>>> Follow this link to subscribe/unsubscribe:
>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>
>>_______________________________________________
>>Powered by www.kitware.com
>>
>>Visit other Kitware open-source projects at
>>http://www.kitware.com/opensource/opensource.html
>>
>>Kitware offers ITK Training Courses, for more information visit:
>>http://kitware.com/products/protraining.php
>>
>>Please keep messages on-topic and check the ITK FAQ at:
>>http://www.itk.org/Wiki/ITK_FAQ
>>
>>Follow this link to subscribe/unsubscribe:
>>http://www.itk.org/mailman/listinfo/insight-developers
>
>
>
> ________________________________
> Notice: This UI Health Care e-mail (including attachments) is covered by the Electronic Communications Privacy Act, 18 U.S.C. 2510-2521, is confidential and may be legally privileged.  If you are not the intended recipient, you are hereby notified that any retention, dissemination, distribution, or copying of this communication is strictly prohibited.  Please reply to the sender that you have received the message in error, then delete it.  Thank you.
> ________________________________



-- 
Unpaid intern in BillsBasement at noware dot com