[CMake] [CTest] Using rerun-failed when tests pass the first time

[Ford Peprah]

Hey all,

My team is using CMake as our build system, and we're also using CTest for
our unit tests.  We wanted to speed up running tests by only running the
tests that failed previously or when the binary file had changed.  While
trying to use '--rerun-failed', we noticed that all our tests were being
run on subsequent ctest invocations, even though they had all passed on the
previous step.  After looking into it, we saw that the file
'LastTestsFailed.log' was not being created.  We tried making a simple
empty one, but that didn't change anything.  We then changed one of our
tests to fail, then tried running 'ctest' with '--rerun-failed' again, and
on the subsequent run it only re-ran that failed test.  We reverted that
test back to normal, and now '--rerun-failed' only runs that test, which
makes sense based on the documentation.

Is this by design?  We expected that if all our tests never failed on the
first invocation, that subsequent runs would not cause those tests to
re-run unless the binary had changed.  The behaviour also seems
inconsistent, since if you add an entry into an empty 'LastTestsFailed.log'
for a non-existant test, then subsequent test runs will not run any of your
tests.  I've given steps below on how to reproduce the issue with the
attached demo.

Version: ctest version 3.7.2

Steps to Reproduce:
 1. Copy over the sources.
 2. Make a build directory, change to that build directory.
 3. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; all
tests should run.
 4. Run 'ctest --rerun-failed' again; all tests should run.
 5. Open up the 'CMakeLists.txt' you copied in the earlier step, uncomment
the division test.
 6. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; all
tests should run.
 7. Run 'ctest --rerun-failed' again; only the 'division' test should run.
 8. Open up the 'CMakeLists.txt' you copied in the earlier step, comment
the division test.
 9. Run 'cmake ..', 'make', then followed by 'ctest --rerun-failed'; should
output "No tests were found!!!"

Expected Behaviour:
 1. If no 'LastTestsFailed.log' exists, should create one; if no tests
failed, then this file should be empty.
 2. On subsequent runs, if a 'LastTestsFailed.log' exists, but is empty,
then no tests should be run.

Attached is the example code (example.zip)
MD5 (example.zip) = 786c191c767f3865b1fc307efe022076


Cheers,
Ford
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20170310/9bb29853/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: example.zip
Type: application/zip
Size: 1708 bytes
Desc: not available
URL: <http://public.kitware.com/pipermail/cmake/attachments/20170310/9bb29853/attachment.zip>