[Insight-developers] Dashboard: Normalizing TIMEOUTS

[Luis Ibanez]

As you may have noticed, the standard practice
of using a single TIMEOUT number for all the
~1,700 test in ITK brings up the challenge of
defining what a good timeout value is for each
machine (and configuration: eg. Release/Debug).

The following proposal was raised in the past,
but we have not acted upon:

1) Add to ITK one or two test that can be considered
    a good benchmark for:

        a) computation power
        b) input / output speed

2) Run those tests and use their timings as a
     base value that characterize this machine.

3)  Define timeout for all tests that are based
     on the values found in (2), multiplied by
     a factor.


Let's say that the computation benchmark takes
2 seconds to run in the machine  foobar.kitware,
then we can tell that the DiffeomorphicDemons
registration test in the same machine should take

           153 x (time of benchmark1 )

(where the number "153" is a factor that we
will have to estimate for each test).

CDash already does a similar thing with the
historical record of the computation time that
it takes to run every test on a given machine,
although this is done on the CDash server,
and therefore it happens too late to be used
as a TIMEOUT mark.

An interesting option as well, could be for
a machine to get access to the historical
record that CDash has computed, and then
use those values as a base for computing
TIMEOUT at the moment of running ctest.


   What do people think of these options ?


         Luis