<HTML>
<HEAD>
<TITLE>Re: [Insight-users] Re: [CMake] ctest multithreaded</TITLE>
</HEAD>
<BODY>
<FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'>Hi all,<BR>
<BR>
As luis pointed out, one of my main concern was the time taken to run the complete test suite before commiting. It painfully slow for the time being, and people that are developping in ITK need to do it almost several times a day. I can not make so many coffee breaks :-D<BR>
<BR>
My basic assumption was indeed that each test was totally independent from any other. The fact that such dependency could not be enforced by any Cmake / ctest command reinforced that assumption.<BR>
<BR>
Yes mathieu, you have no way to be sure that a test run in a given thread will be run before of after a test run in another thread. Note that I’m sending sequenses of tests right now (like 0 to 170, 171 to 340, ...) and thus two consecutive tests should still be consecutive that way (for ITK, the chance that it does not happen are only 7/1012). I think this is not a perfect solution though, as the time needed for each test is not the same. I wanted to do a test where I would send a new test everytime a previous test finished not to be penalyzed by a test that would be much slower. As my simple prototype already reduced the global time to less than I need to go to the coffee machine and come back to my desk, I dropped that idea.<BR>
<BR>
Anyway, the best solution would be to remove those dependency when possible, as luis mentioned. If there were any of those test left, we would need to identify them, and then we could either put them apart, or make the sequences of test in such a way that the dependent tests are run together, in the same thread, in the original order (I suppose the dependency was previously enforced by the order).<BR>
<BR>
David, do you know how many of those tests there is right now in ITK? We could also group them and put them in a single thread to be sure only one “screen analysis” kind of test run at a time, right?<BR>
<BR>
I guess that a huge part of the test suites out there are like this (at least ITK is), and that could be interesting to have a (smart) multithread solution. Smart would also be a way to tell ctest that each test is independent, dependent, or doing screen analysis. As I assume it would break the API and we do not want to do that, we could also maintain separate lists (as text files)...<BR>
<BR>
I would be ready to spend a little bit of time on that. I would appreciate if anyone could help me on the questions I listed in the original e-mail and or give me a hand. It’s kind of (yet another) side project here...<BR>
<BR>
Alex<BR>
<BR>
<BR>
On 7/25/07 5:36 AM, "Luis Ibanez" <luis.ibanez@kitware.com> wrote:<BR>
<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
Hi Gaetan,<BR>
<BR>
I agree that it is an interesting feature to be able to run several tests simultaneously.<BR>
<BR>
It is true that *the large majority* of ITK tests do not depend on the output from <BR>
previous tests. There are however some exceptions, for example, the hole filling <BR>
filters apply their computation on the output of the binary threshold image filter test.<BR>
<BR>
You may get away with the lack of dependencies if you run a non-clean build,<BR>
with the drawback that if test A depends on the output of test B, and yesterday <BR>
test B was broken, then Today's test A will appear as broken, even if the code<BR>
for both A and B is fine today.<BR>
<BR>
Any effort that reduces the computation time of the tests is worth of consideration,<BR>
because that also increases the chances that we will actually run the full testing <BR>
before committing code to the repository ( I say this with the guilt of having <BR>
committed code on Monday without running the full testing...)<BR>
<BR>
The dependencies are probably very few, the challenge is to find them :-/ <BR>
and then to make sure that they are never introduced again. I would guess<BR>
that most of the dependencies can be solved by adding the output file to the<BR>
Data/Input or the Data/Baseline directory, assuming that the file is not too big. <BR>
<BR>
<BR>
Luis<BR>
<BR>
<BR>
<BR>
----------------------------------------------------------------------------------------------------------<BR>
On 7/25/07, <B>Gaëtan Lehmann </B> <gaetan.lehmann@jouy.inra.fr> wrote:<BR>
</SPAN></FONT><BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'>Selon Mathieu Malaterre <mathieu.malaterre@gmail.com>:<BR>
<BR>
> Hi Alex "hanfei",<BR>
><BR>
> I believe there is nothing in the ADD_TEST to declare dependencies. <BR>
> So chance is that you would run a test (depending on output from<BR>
> another) *before* the dependant test is run.<BR>
><BR>
> Is this correct ?<BR>
<BR>
But that's also the case when using the -I option, or any option which allow to <BR>
restrict the set of tests.<BR>
<BR>
The ITK tests for example don't seem to have a dependency on other tests.<BR>
And the itkTestDriver program can now be used to easily run a test program and<BR>
test the output images in a single ADD_TEST() command, so there is no <BR>
dependencies on other tests like with ImageCompare.<BR>
<BR>
That's surely not possible for all the projects, but being able to run the tests<BR>
in parallel as an option (-j ?) would definitively be a nice feature.<BR>
<BR>
><BR>
> -Mathieu<BR>
><BR>
> On 7/24/07, Dr. Alex. GOUAILLARD <hanfei@caltech.edu> wrote:<BR>
> > hi all,<BR>
> ><BR>
> > Annoyed by the time needed by the ITK test suite to run, I made a little <BR>
> > test trying to use ctest in a multithreaded fashion (thanks to the -I<BR>
> > option). Attached is a little test that is yet neither automatic or<BR>
> > complete, but that already give nice results on windows. It could be <BR>
> > directly extended to deal with the memcheck command, which would be<BR>
> > great too (as it is even slower).<BR>
> ><BR>
> > compile it, copy it in your build directory, and it should work fine. <BR>
> ><BR>
> > It first fake running ctest on the first test, for you to read the total<BR>
> > number of tests. It is then prompting you for that number, and launch 8<BR>
> > threads (you can change that number in the code) each running <BR>
> > MaxNumberOfTest / 8 tests. It prints the logs in files named<BR>
> > ctest.log.<ThreadNumber><BR>
> ><BR>
> > pros:<BR>
> > -> On my 8 cores (2 4-cores) machine the gain is indeed a factor 8. 1012 <BR>
> > ITK tests in 5 mn (yum!)<BR>
> ><BR>
> > cons:<BR>
> > -> cmake is trying to write on some temp files, and all but the first<BR>
> > fail to do that.<BR>
> > -> I did not address merging and writing the resulting xml files for <BR>
> > dashboard submission. I need to figure how to redirect each individual<BR>
> > ctest xml output first, then parsing and merging should not really be an<BR>
> > issue.<BR>
> ><BR>
> > questions: <BR>
> > -> how to use ITK framework to make it plateform independent?<BR>
> > -> how to get the number of cores automatically?<BR>
> > -> how to be compliant with the existing 1: writing temp files? <BR>
> > -> how to be compliant with the existant 2: how to redirect the xml output?<BR>
> ><BR>
> > alex "hanfei"<BR>
> ><BR>
> > _______________________________________________<BR>
> > CMake mailing list <BR>
> > CMake@cmake.org<BR>
> > <a href="http://www.cmake.org/mailman/listinfo/cmake">http://www.cmake.org/mailman/listinfo/cmake</a><BR>
> ><BR>
> ><BR>
><BR>
> <BR>
> --<BR>
> Mathieu<BR>
> _______________________________________________<BR>
> Insight-users mailing list<BR>
> Insight-users@itk.org<BR>
> <a href="http://www.itk.org/mailman/listinfo/insight-users">http://www.itk.org/mailman/listinfo/insight-users</a><BR>
><BR>
<BR>
<BR>
_______________________________________________<BR>
CMake mailing list<BR>
CMake@cmake.org<BR>
<a href="http://www.cmake.org/mailman/listinfo/cmake">http://www.cmake.org/mailman/listinfo/cmake</a><BR>
<BR>
</SPAN></FONT></BLOCKQUOTE><FONT FACE="Verdana, Helvetica, Arial"><SPAN STYLE='font-size:12.0px'><BR>
<HR ALIGN=CENTER SIZE="3" WIDTH="95%"></SPAN></FONT><FONT SIZE="2"><FONT FACE="Monaco, Courier New"><SPAN STYLE='font-size:10.0px'>_______________________________________________<BR>
CMake mailing list<BR>
CMake@cmake.org<BR>
<a href="http://www.cmake.org/mailman/listinfo/cmake">http://www.cmake.org/mailman/listinfo/cmake</a><BR>
</SPAN></FONT></FONT></BLOCKQUOTE><FONT SIZE="2"><FONT FACE="Monaco, Courier New"><SPAN STYLE='font-size:10.0px'><BR>
</SPAN></FONT></FONT>
</BODY>
</HTML>