[Insight-users] Re: [CMake] ctest multithreaded

Wed Jul 25 13:29:05 EDT 2007

Hi all,

As luis pointed out, one of my main concern was the time taken to run the
complete test suite before commiting. It painfully slow for the time being,
and people that are developping in ITK need to do it almost several times a
day. I can not make so many coffee breaks :-D

My basic assumption was indeed that each test was totally independent from
any other. The fact that such dependency could not be enforced by any Cmake
/ ctest command reinforced that assumption.

Yes mathieu, you have no way to be sure that a test run in a given thread
will be run before of after a test run in another thread. Note that I¹m
sending sequenses of tests right now (like 0 to 170, 171 to 340, ...) and
thus two consecutive tests should still be consecutive that way (for ITK,
the chance that it does not happen are only 7/1012). I think this is not a
perfect solution though, as the time needed for each test is not the same. I
wanted to do a test where I would send a new test everytime a previous test
finished not to be penalyzed by a test that would be much slower. As my
simple prototype already reduced the global time to less than I need to go
to the coffee machine and come back to my desk, I dropped that idea.

Anyway, the best solution would be to remove those dependency when possible,
as luis mentioned. If there were any of those test left, we would need to
identify them, and then we could either put them apart, or make the
sequences of test in such a way that the dependent tests are run together,
in the same thread, in the original order (I suppose the dependency was
previously enforced by the order).

David, do you know how many of those tests there is right now in ITK? We
could also group them and put them in a single thread to be sure only one
³screen analysis² kind of test run at a time, right?

I guess that a huge part of the test suites out there are like this (at
least ITK is), and that could be interesting to have a (smart) multithread
solution. Smart would also be a way to tell ctest that each test is
independent, dependent, or doing screen analysis. As I assume it would break
the API and we do not want to do that, we could also maintain separate lists
(as text files)...

I would be ready to spend a little bit of time on that. I would appreciate
if anyone could help me on the questions I listed in the original e-mail and
or give me a hand. It¹s kind of (yet another) side project here...

Alex

On 7/25/07 5:36 AM, "Luis Ibanez" <luis.ibanez at kitware.com> wrote:

> 
> Hi Gaetan,
> 
> I agree that it is an interesting feature to be able to run several tests
> simultaneously.
> 
> It is true that *the large majority* of ITK tests do not depend on the output
> from 
> previous tests. There are however some exceptions, for example, the hole
> filling 
> filters apply their computation on the output of the binary threshold image
> filter test.
> 
> You may get away with the lack of dependencies if you run a non-clean build,
> with the drawback that if test A depends on the output of test B, and
> yesterday 
> test B was broken, then Today's test A will appear as broken, even if the code
> for both A and B is fine today.
> 
> Any effort that reduces the computation time of the tests is worth of
> consideration,
> because that also increases the chances that we will actually run the full
> testing 
> before committing code to the repository ( I say this with the guilt of having
> committed code on Monday without running the full testing...)
> 
> The dependencies are probably very few, the challenge is to find them   :-/
> and then to make sure that they are never introduced again.  I would guess
> that most of the dependencies can be solved by adding the output file  to the
> Data/Input or the Data/Baseline directory, assuming that the file is not too
> big. 
> 
> 
>      Luis
> 
> 
> 
> ------------------------------------------------------------------------------
> ----------------------------
> On 7/25/07, Gaëtan Lehmann  <gaetan.lehmann at jouy.inra.fr> wrote:
>> Selon Mathieu Malaterre <mathieu.malaterre at gmail.com>:
>> 
>>> > Hi Alex "hanfei",
>>> >
>>> >   I believe there is nothing in the ADD_TEST to declare dependencies.
>>> > So chance is that you would run a test (depending on output from
>>> > another) *before* the dependant test is run.
>>> >
>>> >   Is this correct ?
>> 
>> But that's also the case when using the -I option, or any option which allow
>> to 
>> restrict the set of tests.
>> 
>> The ITK tests for example don't seem to have a dependency on other tests.
>> And the itkTestDriver program can now be used to easily run a test program
>> and
>> test the output images in a single ADD_TEST() command, so there is no
>> dependencies on other tests like with ImageCompare.
>> 
>> That's surely not possible for all the projects, but being able to run the
>> tests
>> in parallel as an option (-j ?) would definitively be a nice feature.
>> 
>>> >
>>> > -Mathieu
>>> >
>>> > On 7/24/07, Dr. Alex. GOUAILLARD <hanfei at caltech.edu> wrote:
>>>> > > hi all,
>>>> > >
>>>> > > Annoyed by the time needed by the ITK test suite to run, I made a
>>>> little 
>>>> > > test trying to use ctest in a multithreaded fashion (thanks to the -I
>>>> > > option). Attached is a little test that is yet neither automatic or
>>>> > > complete, but that already give nice results on windows. It could be
>>>> > > directly extended to deal with the memcheck command, which would be
>>>> > > great too (as it is even slower).
>>>> > >
>>>> > > compile it, copy it in your build directory, and it should work fine.
>>>> > >
>>>> > > It first fake running ctest on the first test, for you to read the >>>>
total
>>>> > > number of tests. It is then prompting you for that number, and launch 8
>>>> > > threads (you can change that number in the code) each running
>>>> > > MaxNumberOfTest / 8 tests. It prints the logs in files named
>>>> > > ctest.log.<ThreadNumber>
>>>> > >
>>>> > > pros:
>>>> > > -> On my 8 cores (2 4-cores) machine the gain is indeed a factor 8.
>>>> 1012 
>>>> > > ITK tests in 5 mn (yum!)
>>>> > >
>>>> > > cons:
>>>> > > -> cmake is trying to write on some temp files, and all but the first
>>>> > > fail to do that.
>>>> > > -> I did not address merging and writing the resulting xml files for
>>>> > > dashboard submission. I need to figure how to redirect each individual
>>>> > > ctest xml output first, then parsing and merging should not really be
an
>>>> > > issue.
>>>> > >
>>>> > > questions:
>>>> > > -> how to use ITK framework to make it plateform independent?
>>>> > > -> how to get the number of cores automatically?
>>>> > > -> how to be compliant with the existing 1: writing temp files?
>>>> > > -> how to be compliant with the existant 2: how to redirect the xml
>>>> output?
>>>> > >
>>>> > > alex "hanfei"
>>>> > >
>>>> > > _______________________________________________
>>>> > > CMake mailing list
>>>> > > CMake at cmake.org
>>>> > > http://www.cmake.org/mailman/listinfo/cmake
>>>> > >
>>>> > >
>>> >
>>> > 
>>> > --
>>> > Mathieu
>>> > _______________________________________________
>>> > Insight-users mailing list
>>> > Insight-users at itk.org
>>> > http://www.itk.org/mailman/listinfo/insight-users
>>> >
>> 
>> 
>> _______________________________________________
>> CMake mailing list
>> CMake at cmake.org
>> http://www.cmake.org/mailman/listinfo/cmake
>> 
> 
> 
> _______________________________________________
> CMake mailing list
> CMake at cmake.org
> http://www.cmake.org/mailman/listinfo/cmake

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://public.kitware.com/pipermail/cmake/attachments/20070725/5cb400a3/attachment.htm