<div dir="ltr">Hi,<div><br></div><div>I don't that 2 execute_process in sequence in a cmake script could be executed out-of-order but I let CMake developer answer more firmly about that.</div><div>The order of each test created by add_test may vary though.</div><div class="gmail_extra"><br><div class="gmail_quote">2017-05-18 19:32 GMT+02:00 T.Sariyski <span dir="ltr"><<a href="mailto:tzsariysk@gmail.com" target="_blank">tzsariysk@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><div lang="EN-US" link="blue" vlink="purple"><div class="m_6322484952076991974WordSection1"><p class="MsoNormal"><span style="font-size:14.0pt">Hi Eric, <u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">I noticed something that may relates to the problem. Like I said in my previous mail, add_test is a call to cmake:<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"> ADD_TEST(NAME ${TARGET_NAME}<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"> …<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"> COMMAND ${CMAKE_COMMAND} <u></u><u></u></span></p><p class="MsoNormal" style="text-indent:.5in"><span style="font-size:14.0pt">-DPROGRAM1 <u></u><u></u></span></p><p class="MsoNormal" style="text-indent:.5in"><span style="font-size:14.0pt">-DPROGRAM2 <u></u><u></u></span></p><p class="MsoNormal" style="text-indent:.5in"><span style="font-size:14.0pt">…<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Is there a chance that PROGRAM1 and PROGRAM2 are executed asynchronically? Here is what I noticed. If a test is executed from its own binary directory, ‘ctest -VV’, it always pass. If the same test is executed from the main binary directory, ‘ctest -VV -R zamod’, it fails because PROGRAM2 is executed before PROGRAM1 is finished. Attached are the screen outputs for both runs. <u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Thanks,<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Ted<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">==============================<wbr>==============================<wbr>====<u></u><u></u></span></p><p class="MsoNormal"><span lang="FR" style="font-size:14.0pt">cd CTEST/zamod<u></u><u></u></span></p><p class="MsoNormal"><span lang="FR" style="font-size:14.0pt">s1002807:zamod tesari$ ctest -VV<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod/DartConfiguration.<wbr>tcl<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod/DartConfiguration.<wbr>tcl<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Test project /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Constructing a list of tests<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Done constructing a list of tests<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Updating test list for fixtures<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Added 0 tests to meet fixture requirements<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Checking test dependency graph...<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Checking test dependency graph end<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">test 1<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"> Start 1: zamod_serial<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: Test command: /Applications/CMake.app/<wbr>Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=<wbr>true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.<wbr>exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod && /Users/tesari/build/gnu.dbg/<wbr>bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/<wbr>Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts/assert_case.<wbr>py;/Users/tesari/build/gnu.<wbr>dbg/CTEST/zamod/hskdk;/Users/<wbr>tesari/kodiak/trunk/CTEST/<wbr>zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake"<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: Test timeout computed to be: 9.99988e+06<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod && /Users/tesari/build/gnu.dbg/<wbr>bin/kodiak zamod"<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: -- Executing "python;/Users/tesari/kodiak/<wbr>trunk/benchmark/Scripts/<wbr>assert_case.py;/Users/tesari/<wbr>build/gnu.dbg/CTEST/zamod/<wbr>hskdk;/Users/tesari/kodiak/<wbr>trunk/CTEST/zamod/../GOLD/<wbr>zamod/hskdk;0"<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: /Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: Number of arguments: 4 arguments.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: Argument List: ['/Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts/assert_case.<wbr>py', '/Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/<wbr>CTEST/zamod/../GOLD/zamod/<wbr>hskdk', '0']<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: /Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts/assert_case.<wbr>py called with 3 arguments:<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: 1 /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod/hskdk<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: 2 /Users/tesari/kodiak/trunk/<wbr>CTEST/zamod/../GOLD/zamod/<wbr>hskdk<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: 3 0<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: zamod Q0.4 PASS<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: zamod Q0.5 PASS<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1: Kodiak ctest: SUCCESS<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">1/1 Test #1: zamod_serial ..................... Passed 5.73 sec<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">==============================<wbr>==============================<wbr>====<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">s1002807:gnu.dbg tesari$ ctest -VV -R zamod<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/<wbr>DartConfiguration.tcl<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Parse Config file:/Users/tesari/build/gnu.<wbr>dbg/DartConfiguration.tcl<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">UpdateCTestConfiguration from :/Users/tesari/build/gnu.dbg/<wbr>DartConfiguration.tcl<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Parse Config file:/Users/tesari/build/gnu.<wbr>dbg/DartConfiguration.tcl<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Test project /Users/tesari/build/gnu.dbg<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Constructing a list of tests<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Done constructing a list of tests<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Updating test list for fixtures<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Added 0 tests to meet fixture requirements<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Checking test dependency graph...<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">Checking test dependency graph end<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">test 113<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"> Start 113: zamod_serial<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: Test command: /Applications/CMake.app/<wbr>Contents/bin/cmake "-D" "PRE_DELETE_OUTPUT_FILES:BOOL=<wbr>true" "-D" "OUTPUT_FILES:STRING=*kdk*;*.<wbr>exo;*.jpg;*.png;*.log;*~" "-D" "PROGRAM1:STRING=/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod && /Users/tesari/build/gnu.dbg/<wbr>bin/kodiak zamod" "-D" "PROGRAM2:STRING=python;/<wbr>Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts/assert_case.<wbr>py;/Users/tesari/build/gnu.<wbr>dbg/CTEST/zamod/hskdk;/Users/<wbr>tesari/kodiak/trunk/CTEST/<wbr>zamod/../GOLD/zamod/hskdk;0" "-P" "run_and_assert.cmake"<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: Test timeout computed to be: 1500<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: -- Executing "/bin/sh;-c;cd /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod && /Users/tesari/build/gnu.dbg/<wbr>bin/kodiak zamod"<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: -- Executing "python;/Users/tesari/kodiak/<wbr>trunk/benchmark/Scripts/<wbr>assert_case.py;/Users/tesari/<wbr>build/gnu.dbg/CTEST/zamod/<wbr>hskdk;/Users/tesari/kodiak/<wbr>trunk/CTEST/zamod/../GOLD/<wbr>zamod/hskdk;0"<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: /Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: Number of arguments: 4 arguments.<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: Argument List: ['/Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts/assert_case.<wbr>py', '/Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod/hskdk', '/Users/tesari/kodiak/trunk/<wbr>CTEST/zamod/../GOLD/zamod/<wbr>hskdk', '0']<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: /Users/tesari/kodiak/trunk/<wbr>benchmark/Scripts/assert_case.<wbr>py called with 3 arguments:<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: 1 /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod/hskdk<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: 2 /Users/tesari/kodiak/trunk/<wbr>CTEST/zamod/../GOLD/zamod/<wbr>hskdk<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: 3 0<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt">113: /Users/tesari/build/gnu.dbg/<wbr>CTEST/zamod/hskdk does not exist!<u></u><u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><span style="font-size:14.0pt"><u></u> <u></u></span></p><p class="MsoNormal"><b>From:</b> Eric Noulard [<a href="mailto:eric.noulard@gmail.com" target="_blank">mailto:eric.noulard@gmail.com</a><wbr>] <br><b>Sent:</b> Wednesday, May 17, 2017 10:54 PM<br><b>To:</b> T.Sariyski <<a href="mailto:tzsariysk@gmail.com" target="_blank">tzsariysk@gmail.com</a>><br><b>Cc:</b> CMake ML <<a href="mailto:cmake@cmake.org" target="_blank">cmake@cmake.org</a>><br><b>Subject:</b> Re: [CMake] CTest question<u></u><u></u></p><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal">We lack information on how your test are failing. If they all timeout then may be you should setup a proper timeout for the test. The actual execution speed of a test may depend on the load of your machine or the network (depending on what the test are actually doing).<u></u><u></u></p><div><p class="MsoNormal">This may be even more sensible if your test are run in parallel. Do you run ctest with --parallel option? <u></u><u></u></p></div><div><p class="MsoNormal"><u></u> <u></u></p></div><div><p class="MsoNormal">Do your tests depends on each other? If so did you setup test dependencies?<u></u><u></u></p></div></div><div><p class="MsoNormal"><u></u> <u></u></p><div><p class="MsoNormal"><span lang="FR">Le 17 mai 2017 10:59 PM, "T.Sariyski" <</span><a href="mailto:tzsariysk@gmail.com" target="_blank"><span lang="FR">tzsariysk@gmail.com</span></a><span lang="FR">> a écrit :<u></u><u></u></span></p><blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"><div><div><div><p class="MsoNormal"><span style="font-size:14.0pt">Hi,</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:14.0pt"> </span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:14.0pt">I wonder what is the reason that sometimes ctest ends with numerous ‘Failed’ and ‘Timeout’, which ‘ctest –rerun-failed’ reports ‘Passed’. At times the number of these false failures is about one third. I guess that the problem is in the way I set my tests, but I have no clue where to look, or how to ‘debug’ the ctest run.</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:14.0pt">Any advice is highly appreciated.</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:14.0pt"> </span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:14.0pt">Thanks,</span><u></u><u></u></p><p class="MsoNormal"><span style="font-size:14.0pt">Ted</span><u></u><u></u></p></div></div></div><p class="MsoNormal"><br>--<br><br>Powered by <a href="http://www.kitware.com" target="_blank">www.kitware.com</a><br><br>Please keep messages on-topic and check the CMake FAQ at: <a href="http://www.cmake.org/Wiki/CMake_FAQ" target="_blank">http://www.cmake.org/Wiki/<wbr>CMake_FAQ</a><br><br>Kitware offers various services to support the CMake community. For more information on each offering, please visit:<br><br>CMake Support: <a href="http://cmake.org/cmake/help/support.html" target="_blank">http://cmake.org/cmake/help/<wbr>support.html</a><br>CMake Consulting: <a href="http://cmake.org/cmake/help/consulting.html" target="_blank">http://cmake.org/cmake/help/<wbr>consulting.html</a><br>CMake Training Courses: <a href="http://cmake.org/cmake/help/training.html" target="_blank">http://cmake.org/cmake/help/<wbr>training.html</a><br><br>Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" target="_blank">http://www.kitware.com/<wbr>opensource/opensource.html</a><br><br>Follow this link to subscribe/unsubscribe:<br><a href="http://public.kitware.com/mailman/listinfo/cmake" target="_blank">http://public.kitware.com/<wbr>mailman/listinfo/cmake</a><u></u><u></u></p></blockquote></div></div></div></div></div></div><br>--<br>
<br>
Powered by <a href="http://www.kitware.com" rel="noreferrer" target="_blank">www.kitware.com</a><br>
<br>
Please keep messages on-topic and check the CMake FAQ at: <a href="http://www.cmake.org/Wiki/CMake_FAQ" rel="noreferrer" target="_blank">http://www.cmake.org/Wiki/<wbr>CMake_FAQ</a><br>
<br>
Kitware offers various services to support the CMake community. For more information on each offering, please visit:<br>
<br>
CMake Support: <a href="http://cmake.org/cmake/help/support.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/<wbr>support.html</a><br>
CMake Consulting: <a href="http://cmake.org/cmake/help/consulting.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/<wbr>consulting.html</a><br>
CMake Training Courses: <a href="http://cmake.org/cmake/help/training.html" rel="noreferrer" target="_blank">http://cmake.org/cmake/help/<wbr>training.html</a><br>
<br>
Visit other Kitware open-source projects at <a href="http://www.kitware.com/opensource/opensource.html" rel="noreferrer" target="_blank">http://www.kitware.com/<wbr>opensource/opensource.html</a><br>
<br>
Follow this link to subscribe/unsubscribe:<br>
<a href="http://public.kitware.com/mailman/listinfo/cmake" rel="noreferrer" target="_blank">http://public.kitware.com/<wbr>mailman/listinfo/cmake</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Eric<br></div></div></div></div></div>
</div></div>