[Insight-developers] Seeking advice about long running tests

Wed Sep 25 17:20:00 EDT 2013

Matt,
The tree map visualization looks great! A great tool to pick out long
running tests (if the fonts of the label could be scaled accordingly, that
would be even better)  !

Looks like the code coverage loss is not much.
 +1 on your patches.

-Xiaoxiao

On Tue, Sep 24, 2013 at 10:15 PM, Matt McCormick <matt.mccormick at kitware.com
> wrote:

> Hi Brad,
>
> > My ITK valgrind it taking too long. Just some test are just too long.
> How would I disable these tests on a dashboard script?
>
> These tests could be disabled with
>
>   set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)
>
> > Also any idea what the change is coverage is?
>
> Yes, the decrease is about 0.26 %.  Here are my local results:
>
> Default run 1:
>         Covered LOC:         123682
>         Not covered LOC:     22557
>         Total LOC:           146239
>         Percentage Coverage: 84.58%
>
> Default run 2:
>         Covered LOC:         123671
>         Not covered LOC:     22570
>         Total LOC:           146241
>         Percentage Coverage: 84.57%
>
> -LE RUNS_LONG:
>         Covered LOC:         123306
>         Not covered LOC:     22942
>         Total LOC:           146248
>         Percentage Coverage: 84.31%
>
> Thanks,
> Matt
>
>
> > On Sep 20, 2013, at 6:40 PM, Matt McCormick <matt.mccormick at kitware.com>
> wrote:
> >
> >> I did a visualization on test times [1] and it turns out that less
> >> than 1% of the tests (0.97%) take of half (48.7%) of the test times!
> >>
> >> It would be good to add a "RUNS_LONG" label [2] to these tests so that
> >> developers can run the test suite relatively quickly with
> >>
> >>  ctest -LE RUNS_LONG
> >>
> >> and the Visual Studio Debug Nightly builds can exclude them with
> >>
> >>  set(CTEST_TEST_ARGS EXCLUDE_LABEL RUNS_LONG)
> >>
> >> These tests are mostly integration tests that test complicated
> >> algorithms on semi-realistic data.  They have value, but leaving them
> >> out in these situations should not do much harm relative to their time
> >> overhead.
> >>
> >> Thanks,
> >> Matt
> >>
> >> [1] http://review.source.kitware.com/#/c/12733/
> >> [2] http://review.source.kitware.com/#/c/12734/
> >>
> >>
> >> On Fri, Aug 2, 2013 at 11:13 AM, Bill Lorensen <bill.lorensen at gmail.com>
> wrote:
> >>> In general, we do want to make sure the tests produce "correct"
> results and
> >>> make sure there are no regressions. By having the Release build run a
> proper
> >>> set of iterations, the developer can verify the correctness.
> >>>
> >>> That said, using a lower resolution input dataset could probably help
> most
> >>> the these tests and the release/debug times would both be reduced.
> >>>
> >>>
> >>>
> >>> On Fri, Aug 2, 2013 at 10:49 AM, David Cole <dlrdave at aol.com> wrote:
> >>>>
> >>>>
> >>>>> I think it is OK to drop the iterations in the Debug tests. We are
> still
> >>>>> exercising the code. If we
> >>>>> need alternate baselines, we will still catch exceptions. This will
> also
> >>>>> allow more valgrind tests
> >>>>> to run to completion.
> >>>>
> >>>>
> >>>> Ha ha. Ironic. I just noticed the slowest 3 tests have the word
> “Fast” in
> >>>> the test name... Perhaps we should at the very least consider a
> rename. 😊
> >>>>
> >>>> If it’s ok to drop the iterations in the Debug tests, then why not
> drop
> >>>> the iterations in the Release tests, too? There is something to be
> said for
> >>>> the “same” code being run for the test in both Debug and Release. If
> it’s
> >>>> not the same, then I should definitely be able to tell just by
> glancing at
> >>>> the source code for a test that there are Debug/non-Debug
> differences...
> >>>>
> >>>> I would say each individual test should be able to run in *seconds*,
> not
> >>>> minutes or hours. Ideally, less than a second, but I realize that’s
> overly
> >>>> ambitious with some ITK code.
> >>>>
> >>>> However, for the main purpose I have in mind here, namely reducing the
> >>>> time burden on my volunteer machine, I would be absolutely *ecstatic*
> if all
> >>>> of the 20 tests I listed in the original email were able to run in
> under 4
> >>>> or 5 minutes each.
> >>>>
> >>>> Let me know if I can help out in some additional way (beyond just
> >>>> continuing to submit the build to CDash...)
> >>>>
> >>>>
> >>>> Thanks for the discussion, I appreciate it
> >>>> D
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> Unpaid intern in BillsBasement at noware dot com
> >>>
> >>> _______________________________________________
> >>> Powered by www.kitware.com
> >>>
> >>> Visit other Kitware open-source projects at
> >>> http://www.kitware.com/opensource/opensource.html
> >>>
> >>> Kitware offers ITK Training Courses, for more information visit:
> >>> http://kitware.com/products/protraining.php
> >>>
> >>> Please keep messages on-topic and check the ITK FAQ at:
> >>> http://www.itk.org/Wiki/ITK_FAQ
> >>>
> >>> Follow this link to subscribe/unsubscribe:
> >>> http://www.itk.org/mailman/listinfo/insight-developers
> >>>
> >> _______________________________________________
> >> Powered by www.kitware.com
> >>
> >> Visit other Kitware open-source projects at
> >> http://www.kitware.com/opensource/opensource.html
> >>
> >> Kitware offers ITK Training Courses, for more information visit:
> >> http://kitware.com/products/protraining.php
> >>
> >> Please keep messages on-topic and check the ITK FAQ at:
> >> http://www.itk.org/Wiki/ITK_FAQ
> >>
> >> Follow this link to subscribe/unsubscribe:
> >> http://www.itk.org/mailman/listinfo/insight-developers
> >
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.php
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
>

-- 

---------------------------------------------
*Xiaoxiao Liu*, Ph.D.
R & D Engineer
Kitware Inc <http://www.kitware.com/>.
Clifton Park, NY
Phone: (518) 881-4924  or  (518) 371-3971 x124
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.itk.org/pipermail/insight-developers/attachments/20130925/615d8d96/attachment.htm>