[Insight-developers] [Cdash] Dashboard: Normalizing TIMEOUTS

Luis Ibanez luis.ibanez at kitware.com
Wed Jan 20 14:12:58 EST 2010


Hi Julien,


I added a feature request entry
to the MANTIS bug tracker:

http://public.kitware.com/Bug/view.php?id=10170


     Thanks


              Luis


---------------------------------------------------------
On Wed, Jan 20, 2010 at 10:14 AM, Julien Jomier
<julien.jomier at kitware.com> wrote:
> Luis,
>
> I like the idea very much and it seems that the work Zack (in CC) did for
> running CTest in parallel might help. Basically newer version of CTest
> generates a file with the time each test took to run, then on the next
> submission CTest uses the times to schedule the parallel runs efficiently.
>
> The plan in the future is to have CTest download the test time history from
> CDash which is consistent with your idea. I think it would be good to log a
> feature request (if not already documented) either in CMake or CDash bug
> tracker and maybe point to a wiki page.
>
> Julien
>
> On 1/19/2010 8:31 PM, Luis Ibanez wrote:
>>
>> Hi Brad,
>>
>>
>> A) The problem:
>>
>>
>>       Maintainer of dashboard machines do not have a
>>       clear reference for how to set up their TIMEOUT
>>       values.
>>
>>       Is 10 minutes the right timeout value ?
>>       or is it 20 minutes ? or 2 minutes ?
>>
>>       Of course it all depends of the machine capabilities,
>>       (number of cores, RAM, disk speed), the OS, the
>>       compiler, and the compilation flags used (e.g. Debug
>>       Release...)
>>
>>
>>      If they under-estimate, many test may fail just because
>>      they are not allowed to run long enough for them to complete.
>>      As it was happening in some RogueResearch machines.
>>
>>      If they over-estimate, then their machines may get to
>>      waste time running tests that are trapped in infinite
>>      loops (like the itkStatisticsAlgorithmTests in llvm), or
>>      that take unrealistic times to complete, like running the
>>      large-memory write test in a machine that has low RAM
>>      and end up swapping to disk.
>>
>>
>> B)  A single TIMEOUT value for all test doesn't represents
>>       the very large range of time that different individual test
>>       in ITK require.
>>
>>       E.g. some test are expected to finish in 0.1 seconds
>>       while others require 20 minutes.
>>
>>       However, we control them all with a single TIMEOUT.
>>
>>       One size for all 1,742 different tests...
>>
>>
>> C)   I apologize for not having explained the idea clearly
>>        enough. I'm not suggesting that for anyone to "manually"
>>        define timeout factors for the tests.   There is no need to
>>        do this manually. The process can certainly be automated.
>>
>>        As you pointed out, one possible mechanism is to simply
>>        harvest the statistical data that CDash has already stored
>>        for every machine that contributes to the Dashboard.
>>
>>        CDash already computes a mean and standard deviation
>>        for the amount of time that it take to run every test, and this
>>        is done independently for every machine that contributes
>>        to the Dashboard. This is how CDash can evaluate and
>>        report TIMING failures per tests.
>>
>>        We could simply use that information in order to compute
>>        better-adjusted timeouts on a test-by-test basis.
>>
>>
>>
>>       Luis
>>
>>
>> -----------------------------------------------------------
>> On Tue, Jan 19, 2010 at 1:26 PM, Bradley Lowekamp
>> <blowekamp at mail.nih.gov>  wrote:
>>>
>>> Hello Luis,
>>>
>>> Why? What is wrong with the current system and what are we trying to fix
>>> or accomplish?
>>>
>>> Having to manually define timeouts for all or even many tests sounds like
>>> a lot of maintenance, and it should be avoided. It would only work if it
>>> could be done automatically. It could be done as a batch analysis of the
>>> cdash records to solve the linear system of equations proposed by Luis.
>>>
>>> There are already some existing resources used in testing the could be
>>> better automatically handled. The memory used of the program, and the number
>>> of threads. These can play into the number of parallel tests that can be
>>> run. Just last week, my system with 32GB, was running out of memory running
>>> ctest in parallel.
>>>
>>> Brad
>>>
>>> On Jan 18, 2010, at 11:32 AM, Luis Ibanez wrote:
>>>
>>>> As you may have noticed, the standard practice
>>>> of using a single TIMEOUT number for all the
>>>> ~1,700 test in ITK brings up the challenge of
>>>> defining what a good timeout value is for each
>>>> machine (and configuration: eg. Release/Debug).
>>>>
>>>> The following proposal was raised in the past,
>>>> but we have not acted upon:
>>>>
>>>> 1) Add to ITK one or two test that can be considered
>>>>    a good benchmark for:
>>>>
>>>>        a) computation power
>>>>        b) input / output speed
>>>>
>>>> 2) Run those tests and use their timings as a
>>>>     base value that characterize this machine.
>>>>
>>>> 3)  Define timeout for all tests that are based
>>>>     on the values found in (2), multiplied by
>>>>     a factor.
>>>>
>>>>
>>>> Let's say that the computation benchmark takes
>>>> 2 seconds to run in the machine  foobar.kitware,
>>>> then we can tell that the DiffeomorphicDemons
>>>> registration test in the same machine should take
>>>>
>>>>           153 x (time of benchmark1 )
>>>>
>>>> (where the number "153" is a factor that we
>>>> will have to estimate for each test).
>>>>
>>>> CDash already does a similar thing with the
>>>> historical record of the computation time that
>>>> it takes to run every test on a given machine,
>>>> although this is done on the CDash server,
>>>> and therefore it happens too late to be used
>>>> as a TIMEOUT mark.
>>>>
>>>> An interesting option as well, could be for
>>>> a machine to get access to the historical
>>>> record that CDash has computed, and then
>>>> use those values as a base for computing
>>>> TIMEOUT at the moment of running ctest.
>>>>
>>>>
>>>>   What do people think of these options ?
>>>>
>>>>
>>>>         Luis
>>>> _______________________________________________
>>>> Powered by www.kitware.com
>>>>
>>>> Visit other Kitware open-source projects at
>>>> http://www.kitware.com/opensource/opensource.html
>>>>
>>>> Kitware offers ITK Training Courses, for more information visit:
>>>> http://kitware.com/products/protraining.html
>>>>
>>>> Please keep messages on-topic and check the ITK FAQ at:
>>>> http://www.itk.org/Wiki/ITK_FAQ
>>>>
>>>> Follow this link to subscribe/unsubscribe:
>>>> http://www.itk.org/mailman/listinfo/insight-developers
>>>
>>>
>> _______________________________________________
>> Cdash mailing list
>> Cdash at public.kitware.com
>> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>>
>


More information about the Insight-developers mailing list