[CMake] CxxTest failure output running unit tests

[Philip Lowman]

...... Original Message .......
On Mon, 08 Dec 2008 10:50:28 -0500 "Bill Hoffman" 
<bill.hoffman at kitware.com> wrote:
>Philip Lowman wrote:
>> On Sun, Dec 7, 2008 at 5:44 PM, Tron Thomas <tron.thomas at verizon.net 
>> <mailto:tron.thomas at verizon.net>> wrote:
>> 
>>     It would be nice if the original "test" and RUN_TESTS targets could
>>     be configured this way without having to create and additional
>>     target that replaces them.
>> 
>> 
>> Patch attached here:
>> http://public.kitware.com/Bug/view.php?id=8255
>> 
>> Here's a start at this that doesn't break existing behavior.  If the 
>> user specifies a "--output-on-failure" flag to CTest or defines the 
>> environment variable CTEST_OUTPUT_ON_FAILURE then when a test fails, 
>> CTest will output whatever the test program outputted to the screen.  
>> Usually this should be the test assertion(s) that failed.  Here's an 
>> example of what it might look like when using CxxTest.
>> 
>> user at ubuntu:~/src/test_cmake_cxxtest/build$ echo "export 
>> CTEST_OUTPUT_ON_FAILURE=1" >> ~/.bashrc
>> user at ubuntu:~/src/test_cmake_cxxtest/build$ source ~/.bashrc
>> user at ubuntu:~/src/test_cmake_cxxtest/build$ make test
>> Running tests...
>> Start processing tests
>> Test project /home/user/src/test_cmake_cxxtest/build
>>   1/  3 Testing unittest_addition ............   Passed
>>   2/  3 Testing unittest_addition_fail .......
>> *** Test Failed:
>> Running 1 test
>> In MyTestSuite::testAddition:
>> /home/user/src/test_cmake_cxxtest/addition_failure.h:8: Error: Expected 
>> (1 + 1 == 3), found (2 != 3)
>> Failed 1 of 1 test
>> Success rate: 0%
>> 
>> ***Failed
>>   3/  3 Testing unittest_subtraction .........   Passed
>> 
>> 67% tests passed, 1 tests failed out of 3
>> 
>> The following tests FAILED:
>>       2 - unittest_addition_fail (Failed)
>> Errors while running CTest
>> make: *** [test] Error 8
>> 
>
>I still think it looks ugly and hard to see the tests that failed.....

Not all testing suites are going to be as verbose as CxxTest.  It's also not going to be any 
easier on the eyes in the verbose logfile than it is on stdout.  In fact I would argue it's far 
more readable on stdout because with the verbose option there is a lot more logging.

Furthermore, the possibility remains that CxxTest  may be able to be 
customized to simplify the output when running within another testing 
system.

>
>Does the  Testing/Temporary/LastTest.log file help at all?   So, if you 
>run tests and there are failures, you can look at that file to see the 
>detail.  Maybe if ctest did something like this:
>The following tests FAILED:
>        2 - unittest_addition_fail (Failed)
>(See Testing/Temporary/LastTest.log for details.)

LastTest.log is a nice thing to have but if the user has 800 unit tests I 
think he'd prefer not to look through the entire logfile.

How about giving the user the command to run to get information for the 
tests that failed?

(Run "ctest --output-on-failure -N 2,5,9-11" for details about the failed 
tests.) 

Obviously -N would also need to be implemented for this to work right (-I 
is kinda bloated and doesn't support multiple ranges).

-- 
Philip Lowman