[CMake] tests running out of order on cmake

[Michael Wild]

Yes, add_test() should probably mention set_tests_properties(). However,
I don't think that any of the set_xxx_properties() mention the
corresponding "PROPERTIES ON XXX" section, which arguably would be a
very good idea.

Feel free to submit a patch, I'm sure the CMake-developers will gladly
accept it ;-)

Michael


On 06/28/2011 05:01 PM, j s wrote:
> On Tue, Jun 28, 2011 at 9:57 AM, Michael Wild <themiwi at gmail.com> wrote:
>> I agree that the information is not very discoverable if you've only
>> read the doc about add_test(), but it is there:
>>
>> http://cmake.org/cmake/help/cmake-2-8-docs.html#command:set_tests_properties
>>
>> http://cmake.org/cmake/help/cmake-2-8-docs.html#section_PropertiesonTests
> 
> So the bug in the documentation is that:
> 
> add_test does not mention set_test_properties
> 
> set_test_properties documents only a few properties, but makes no
> mention of the "PROPERTIES ON TESTS" section.
> 
> Another bug in the software, is that there is no way to specify that
> the tests should run in order.  I tried the ctest -I option and it
> didn't work.
> 
> Juan
> 
> 
>>
>> Michael
>>
>> On 06/28/2011 04:52 PM, j s wrote:
>>> On Tue, Jun 28, 2011 at 9:46 AM, j s <j.s4403 at gmail.com> wrote:
>>>> On Tue, Jun 28, 2011 at 9:33 AM, Michael Wild <themiwi at gmail.com> wrote:
>>>>> If the tests depend on each other, you *must* tell CTest so, by setting
>>>>> the DEPENDS property using the set_tests_properties() function.
>>>>>
>>>>> Normally, running tests in arbitrary order is considered to be a
>>>>> feature, helping you to not to introduce accidental dependencies.
>>>>
>>>> I respectfully disagree.  It is a bug, which had been marked as major
>>>> severity, as noted by the link in my previous email.  I have 85
>>>> regressions, and the first ones run the fastest, so I immediately know
>>>> if there is a problem.  Now that I am aware of set_test_properties,
>>>> I'll set them for the tests that need them.  When not running tests in
>>>> parallel, which is always, they need to run in the exact order
>>>> specified.  It is only a feature, if there is a default option like
>>>> schedule_sequential, which allows the program to work as I had come to
>>>> expect over the past 6 years.
>>>
>>> It is also a bug since the man page for cmake 2.8.4 does not mention
>>> setting dependencies with set_test_properties.
>>>
>>>>
>>>> Juan
>>>>
>>>>>
>>>>> Michael
>>>>>
>>>>> On 06/28/2011 04:16 PM, j s wrote:
>>>>>> Can someone please let me know how to get the Mac OS X version of
>>>>>> cmake 2.8.4 to run the tests in the order in which they are added?
>>>>>> For some bizarre reason they are running out of order:
>>>>>>
>>>>>> Sample output (without actual test name).
>>>>>>  1/85 Test #64: XXXXXXXXXXXXXXXX .............***Failed    0.01 sec
>>>>>>       Start 63: XXXXXXXXXXXXXXXX
>>>>>>  2/85 Test #63: XXXXXXXXXXXXXXXX..............   Passed    0.00 sec
>>>>>>       Start 62: XXXXXXXXXXXXXXXX
>>>>>>  3/85 Test #62: XXXXXXXXXXXXXXXX..............   Passed    0.00 sec
>>>>>>       Start 61: XXXXXXXXXXXXXXXX
>>>>>>
>>>>>> This is preventing from verifying my recent changes on the mac.  The
>>>>>> Ubuntu linux version does not have this issue.
>>>>>>
>>>>>> Juan
>>>>>>
>>>>>>
>>>>>> On Sat, Jun 25, 2011 at 2:51 PM, j s <j.s4403 at gmail.com> wrote:
>>>>>>> Unfortunately the problem came back.  I there a command line option to
>>>>>>> fix this out of order execution feature.
>>>>>>>
>>>>>>> Juan
>>>>>>>
>>>>>>> On Sat, Jun 25, 2011 at 1:30 PM, j s <j.s4403 at gmail.com> wrote:
>>>>>>>> I was able to update the cmake dmg from the website, and now my tests
>>>>>>>> are running in the proper order.
>>>>>>>>
>>>>>>>> On Sat, Jun 25, 2011 at 1:23 PM, j s <j.s4403 at gmail.com> wrote:
>>>>>>>>> How do I tell the macports version of cmake 2.8.1  on the mac to run
>>>>>>>>> the tests in exactly the order they were originally specified?  I'm in
>>>>>>>>> the process of updating macports right now to see if it goes back to
>>>>>>>>> the sequential default in later versions?  man ctest reveals
>>>>>>>>> --schedule-random, but no sequential option.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Juan