[Insight-developers] Dashboard: Normalizing TIMEOUTS

[Bradley Lowekamp]

Hello Luis,

Why? What is wrong with the current system and what are we trying to fix or accomplish?

Having to manually define timeouts for all or even many tests sounds like a lot of maintenance, and it should be avoided. It would only work if it could be done automatically. It could be done as a batch analysis of the cdash records to solve the linear system of equations proposed by Luis.

There are already some existing resources used in testing the could be better automatically handled. The memory used of the program, and the number of threads. These can play into the number of parallel tests that can be run. Just last week, my system with 32GB, was running out of memory running ctest in parallel. 

Brad

On Jan 18, 2010, at 11:32 AM, Luis Ibanez wrote:

> As you may have noticed, the standard practice
> of using a single TIMEOUT number for all the
> ~1,700 test in ITK brings up the challenge of
> defining what a good timeout value is for each
> machine (and configuration: eg. Release/Debug).
> 
> The following proposal was raised in the past,
> but we have not acted upon:
> 
> 1) Add to ITK one or two test that can be considered
>    a good benchmark for:
> 
>        a) computation power
>        b) input / output speed
> 
> 2) Run those tests and use their timings as a
>     base value that characterize this machine.
> 
> 3)  Define timeout for all tests that are based
>     on the values found in (2), multiplied by
>     a factor.
> 
> 
> Let's say that the computation benchmark takes
> 2 seconds to run in the machine  foobar.kitware,
> then we can tell that the DiffeomorphicDemons
> registration test in the same machine should take
> 
>           153 x (time of benchmark1 )
> 
> (where the number "153" is a factor that we
> will have to estimate for each test).
> 
> CDash already does a similar thing with the
> historical record of the computation time that
> it takes to run every test on a given machine,
> although this is done on the CDash server,
> and therefore it happens too late to be used
> as a TIMEOUT mark.
> 
> An interesting option as well, could be for
> a machine to get access to the historical
> record that CDash has computed, and then
> use those values as a base for computing
> TIMEOUT at the moment of running ctest.
> 
> 
>   What do people think of these options ?
> 
> 
>         Luis
> _______________________________________________
> Powered by www.kitware.com
> 
> Visit other Kitware open-source projects at
> http://www.kitware.com/opensource/opensource.html
> 
> Kitware offers ITK Training Courses, for more information visit:
> http://kitware.com/products/protraining.html
> 
> Please keep messages on-topic and check the ITK FAQ at:
> http://www.itk.org/Wiki/ITK_FAQ
> 
> Follow this link to subscribe/unsubscribe:
> http://www.itk.org/mailman/listinfo/insight-developers