[Insight-developers] Dashboard: Normalizing TIMEOUTS

[Bill Lorensen]

Luis,

It would need to be broken down by build type. For example a debug
build on some platforms can be 10-30 slower that a release build.

Bill

On Mon, Jan 18, 2010 at 11:32 AM, Luis Ibanez <luis.ibanez at kitware.com> wrote:
> As you may have noticed, the standard practice
> of using a single TIMEOUT number for all the
> ~1,700 test in ITK brings up the challenge of
> defining what a good timeout value is for each
> machine (and configuration: eg. Release/Debug).
>
> The following proposal was raised in the past,
> but we have not acted upon:
>
> 1) Add to ITK one or two test that can be considered
>    a good benchmark for:
>
>        a) computation power
>        b) input / output speed
>
> 2) Run those tests and use their timings as a
>     base value that characterize this machine.
>
> 3)  Define timeout for all tests that are based
>     on the values found in (2), multiplied by
>     a factor.
>
>
> Let's say that the computation benchmark takes
> 2 seconds to run in the machine  foobar.kitware,
> then we can tell that the DiffeomorphicDemons
> registration test in the same machine should take
>
>           153 x (time of benchmark1 )
>
> (where the number "153" is a factor that we
> will have to estimate for each test).
>
> CDash already does a similar thing with the
> historical record of the computation time that
> it takes to run every test on a given machine,
> although this is done on the CDash server,
> and therefore it happens too late to be used
> as a TIMEOUT mark.
>
> An interesting option as well, could be for
> a machine to get access to the historical
> record that CDash has computed, and then
> use those values as a base for computing
> TIMEOUT at the moment of running ctest.
>
>
>   What do people think of these options ?
>
>
>         Luis
> _______________________________________________
> Powered by www.kitware.com
>
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
>
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
>
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
>
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers
>