[Cdash] [Insight-developers] Dashboard: Normalizing TIMEOUTS
David Cole
david.cole at kitware.com
Wed Jan 20 19:35:54 UTC 2010
I just added a note to that bug report.
There is already *a* way to do it, although perhaps not optimal, that can be
demonstrated right now with CDash 1.6... You'll be able to try this on the
ITK CDash database next week. The following link is a test from the VTKEdge
project on www.kitware.com/CDash.
There is an easy way to gather this data by hand in the forthcoming CDash
1.6 using the "queryTests.php" page and filters. The following hyperlink
shows the passing test results for a given test in VTKEdge on a given
site/buildname combination for the last 30 days... :-)
https://www.kitware.com/CDash/queryTests.php?project=VTKEdge&filtercount=5&showfilters=1&filtercombine=and&field1=testname/string&compare1=61&value1=PaintbrushTest2&field2=status/string&compare2=61&value2=Passed&field3=site/string&compare3=61&value3=Cairparavel.kitware&field4=buildname/string&compare4=61&value4=all-release-VS8&field5=buildstarttime/date&compare5=83&value5=30%20days%20ago[
^<https://www.kitware.com/CDash/queryTests.php?project=VTKEdge&filtercount=5&showfilters=1&filtercombine=and&field1=testname/string&compare1=61&value1=PaintbrushTest2&field2=status/string&compare2=61&value2=Passed&field3=site/string&compare3=61&value3=Cairparavel.kitware&field4=buildname/string&compare4=61&value4=all-release-VS8&field5=buildstarttime/date&compare5=83&value5=30%20days%20ago>
]
It may or may not be easily machine readable in this format, but you can
easily scan it as a human being and get a fairly good idea about the timing
of the test on that build for the number of days specified...
Cheers,
David C.
On Wed, Jan 20, 2010 at 2:12 PM, Luis Ibanez <luis.ibanez at kitware.com>wrote:
> Hi Julien,
>
>
> I added a feature request entry
> to the MANTIS bug tracker:
>
> http://public.kitware.com/Bug/view.php?id=10170
>
>
> Thanks
>
>
> Luis
>
>
> ---------------------------------------------------------
> On Wed, Jan 20, 2010 at 10:14 AM, Julien Jomier
> <julien.jomier at kitware.com> wrote:
> > Luis,
> >
> > I like the idea very much and it seems that the work Zack (in CC) did for
> > running CTest in parallel might help. Basically newer version of CTest
> > generates a file with the time each test took to run, then on the next
> > submission CTest uses the times to schedule the parallel runs
> efficiently.
> >
> > The plan in the future is to have CTest download the test time history
> from
> > CDash which is consistent with your idea. I think it would be good to log
> a
> > feature request (if not already documented) either in CMake or CDash bug
> > tracker and maybe point to a wiki page.
> >
> > Julien
> >
> > On 1/19/2010 8:31 PM, Luis Ibanez wrote:
> >>
> >> Hi Brad,
> >>
> >>
> >> A) The problem:
> >>
> >>
> >> Maintainer of dashboard machines do not have a
> >> clear reference for how to set up their TIMEOUT
> >> values.
> >>
> >> Is 10 minutes the right timeout value ?
> >> or is it 20 minutes ? or 2 minutes ?
> >>
> >> Of course it all depends of the machine capabilities,
> >> (number of cores, RAM, disk speed), the OS, the
> >> compiler, and the compilation flags used (e.g. Debug
> >> Release...)
> >>
> >>
> >> If they under-estimate, many test may fail just because
> >> they are not allowed to run long enough for them to complete.
> >> As it was happening in some RogueResearch machines.
> >>
> >> If they over-estimate, then their machines may get to
> >> waste time running tests that are trapped in infinite
> >> loops (like the itkStatisticsAlgorithmTests in llvm), or
> >> that take unrealistic times to complete, like running the
> >> large-memory write test in a machine that has low RAM
> >> and end up swapping to disk.
> >>
> >>
> >> B) A single TIMEOUT value for all test doesn't represents
> >> the very large range of time that different individual test
> >> in ITK require.
> >>
> >> E.g. some test are expected to finish in 0.1 seconds
> >> while others require 20 minutes.
> >>
> >> However, we control them all with a single TIMEOUT.
> >>
> >> One size for all 1,742 different tests...
> >>
> >>
> >> C) I apologize for not having explained the idea clearly
> >> enough. I'm not suggesting that for anyone to "manually"
> >> define timeout factors for the tests. There is no need to
> >> do this manually. The process can certainly be automated.
> >>
> >> As you pointed out, one possible mechanism is to simply
> >> harvest the statistical data that CDash has already stored
> >> for every machine that contributes to the Dashboard.
> >>
> >> CDash already computes a mean and standard deviation
> >> for the amount of time that it take to run every test, and this
> >> is done independently for every machine that contributes
> >> to the Dashboard. This is how CDash can evaluate and
> >> report TIMING failures per tests.
> >>
> >> We could simply use that information in order to compute
> >> better-adjusted timeouts on a test-by-test basis.
> >>
> >>
> >>
> >> Luis
> >>
> >>
> >> -----------------------------------------------------------
> >> On Tue, Jan 19, 2010 at 1:26 PM, Bradley Lowekamp
> >> <blowekamp at mail.nih.gov> wrote:
> >>>
> >>> Hello Luis,
> >>>
> >>> Why? What is wrong with the current system and what are we trying to
> fix
> >>> or accomplish?
> >>>
> >>> Having to manually define timeouts for all or even many tests sounds
> like
> >>> a lot of maintenance, and it should be avoided. It would only work if
> it
> >>> could be done automatically. It could be done as a batch analysis of
> the
> >>> cdash records to solve the linear system of equations proposed by Luis.
> >>>
> >>> There are already some existing resources used in testing the could be
> >>> better automatically handled. The memory used of the program, and the
> number
> >>> of threads. These can play into the number of parallel tests that can
> be
> >>> run. Just last week, my system with 32GB, was running out of memory
> running
> >>> ctest in parallel.
> >>>
> >>> Brad
> >>>
> >>> On Jan 18, 2010, at 11:32 AM, Luis Ibanez wrote:
> >>>
> >>>> As you may have noticed, the standard practice
> >>>> of using a single TIMEOUT number for all the
> >>>> ~1,700 test in ITK brings up the challenge of
> >>>> defining what a good timeout value is for each
> >>>> machine (and configuration: eg. Release/Debug).
> >>>>
> >>>> The following proposal was raised in the past,
> >>>> but we have not acted upon:
> >>>>
> >>>> 1) Add to ITK one or two test that can be considered
> >>>> a good benchmark for:
> >>>>
> >>>> a) computation power
> >>>> b) input / output speed
> >>>>
> >>>> 2) Run those tests and use their timings as a
> >>>> base value that characterize this machine.
> >>>>
> >>>> 3) Define timeout for all tests that are based
> >>>> on the values found in (2), multiplied by
> >>>> a factor.
> >>>>
> >>>>
> >>>> Let's say that the computation benchmark takes
> >>>> 2 seconds to run in the machine foobar.kitware,
> >>>> then we can tell that the DiffeomorphicDemons
> >>>> registration test in the same machine should take
> >>>>
> >>>> 153 x (time of benchmark1 )
> >>>>
> >>>> (where the number "153" is a factor that we
> >>>> will have to estimate for each test).
> >>>>
> >>>> CDash already does a similar thing with the
> >>>> historical record of the computation time that
> >>>> it takes to run every test on a given machine,
> >>>> although this is done on the CDash server,
> >>>> and therefore it happens too late to be used
> >>>> as a TIMEOUT mark.
> >>>>
> >>>> An interesting option as well, could be for
> >>>> a machine to get access to the historical
> >>>> record that CDash has computed, and then
> >>>> use those values as a base for computing
> >>>> TIMEOUT at the moment of running ctest.
> >>>>
> >>>>
> >>>> What do people think of these options ?
> >>>>
> >>>>
> >>>> Luis
> >>>> _______________________________________________
> >>>> Powered by www.kitware.com
> >>>>
> >>>> Visit other Kitware open-source projects at
> >>>> http://www.kitware.com/opensource/opensource.html
> >>>>
> >>>> Kitware offers ITK Training Courses, for more information visit:
> >>>> http://kitware.com/products/protraining.html
> >>>>
> >>>> Please keep messages on-topic and check the ITK FAQ at:
> >>>> http://www.itk.org/Wiki/ITK_FAQ
> >>>>
> >>>> Follow this link to subscribe/unsubscribe:
> >>>> http://www.itk.org/mailman/listinfo/insight-developers
> >>>
> >>>
> >> _______________________________________________
> >> Cdash mailing list
> >> Cdash at public.kitware.com
> >> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
> >>
> >
> _______________________________________________
> Cdash mailing list
> Cdash at public.kitware.com
> http://public.kitware.com/cgi-bin/mailman/listinfo/cdash
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cdash/attachments/20100120/a8f747e1/attachment-0003.htm>
More information about the CDash
mailing list