[CMake] Using CTest with a different dashboard

[Michael Hertling]

On 11/09/2011 06:06 PM, EXT-York, Gantry wrote:
> I'm looking for suggestions from someone who has used CTest to report results to a dashboard other than CDash.
> 
> I'd like to be able to simply type
> 
> gmake test TESTOPTS="-report"
> 
> then have CTest traverse all the directories and run the defined tests.  Not only would I like it to run these tests, but I'd like it to run it through a wrapper script that I use to report the results to a different dashboard.
> 
> I've tried this with
> 
> set( TEST_WRAPPER /full/path/to/test_wrapper )
> ...
> add_test( unit_test1 )                                                                                  # syntax 1
> add_test( /full/path/to/test_wrapper --report unit_test2 )        # syntax 2
> add_test( ${TEST_WRAPPER} ${TESTOPTS} unit_test3 )                # syntax 3
> 
> There are several problems I've noticed with this:
> 
> Syntax 1 works fine.  It knows where the name unit_test1 is located and runs, but since it isn't run through the wrapper script, nothing is reported to our dashboard.

Actually, this can't work as it misses the test's name. Moreover, you
should use the more powerful ADD_TEST(NAME ... COMMAND ...) signature.

> Syntax 2 works does not work.  All the arguments are quoted.  "-report" fails as an option because of the quotes, and "unit_test2" is treated as a string not as a name of a CMake object with a location.

Use generator expressions to refer to a target as a parameter for a test:

ADD_TEST(... COMMAND path/to/wrapper --report $<TARGET_FILE:unit_test2>)

AFAIK, CMake recognizes a target only as the command's first argument.

> Syntax 3 is just a variation of syntax 2 and also fails, but it is preferable to define the wrapper script in the highest level CMakeLists.txt file and have that value carried through to sub directories.

Absolutely, and if the script is not contained in or generated by your
project, use FIND_PROGRAM() to locate it and to populate the variable.

> So I tried this
> 
> set( TEST_WRAPPER /full/path/to/test_wrapper_report )
> ...
> add_test( ${TEST_WRAPPER_REPORT} "${CURRENT_DIR}/ unit_test4" )                # syntax 4
> 
> I modified the test wrapper to report without having to specify the -report option.
> 
> In this case, everything seemed to work, but the return/exit value coming back from the wrapper script was wrong.  It seems that something is going on in the CTest framework that changes this value.  Instead of getting a value like -1, 0, 1, I was getting a value like 32844.  I verified that unit_test4 returned the correct return status to the wrapper script and the wrapper script returned the correct return status when run outside of the CTest framework.

Usually, it should be possible to specify any arguments to a test
driver as arguments behind ADD_TEST()'s COMMAND clause, and there
should also be no issues w.r.t. quotes. Thus, you might try anew
with generator expressions and correctly set up ADD_TEST()
commands, and report if it still doesn't work.

Regards,

Michael