[CMake] making Nightly builds easier to setup

[Martin Apel]

Alexander Neundorf wrote:
> On Monday 10 November 2008, Martin Apel wrote:
> ...
>   
>> I recently played around with nightly builds as well. I used to have a
>> setup for experimental builds, but never could get the svn checkout to
>> run. With the approach described above, I was finally able
>> to run checkout from svn from within ctest. Unfortunately it seems that
>> some variables are not used anymore with this approach of generating
>> builds. I found that CTEST_INITIAL_CACHE 
>>     
>
> I plan to do something about this.
>
>   
>> as well as CTEST_ENVIRONMENT seem
>>     
>
> This one probably too.
>
> Which other problems do you have in detail ?
>   
Hi Alex,

thanks a lot for your initiative to improve the situation with CTest.
For the CTEST_INITIAL_CACHE issue I used 'FILE WRITE' as recommended by
David, which works fine now.
There are three remaining problems with ctest:
1. Memory checking using valgrind
2. Coverage testing using gcov
3. Setting the test timeout correctly
4. Excluding certain tests during memchecking

My old setup looked like this:

SET (CTEST_INITIAL_CACHE "
MAKECOMMAND:STRING=make -i -j3 world
DART_TESTING_TIMEOUT:STRING=3600
CMAKE_CXX_FLAGS_DEBUG:STRING=-g -fprofile-arcs -ftest-coverage
CMAKE_EXE_LINKER_FLAGS_DEBUG:STRING=-g -fprofile-arcs -ftest-coverage
CMAKE_SHARED_LINKER_FLAGS_DEBUG:STRING=-g -fprofile-arcs -ftest-coverage
COVERAGE_COMMAND:STRING=gcov-4.2
")

SET (CTEST_CMAKE_COMMAND  "cmake -DCMAKE_BUILD_TYPE=Debug")
SET (CTEST_COMMAND      
     "ctest -D Experimental"
     "ctest -D ExperimentalMemCheck -E Compare"
     "ctest -D ExperimentalSubmit"
     )

For the new script I tried something along these lines:
SET (CTEST_MEMORYCHECK_COMMAND "valgrind")
SET (CTEST_MEMORYCHECK_COMMAND_OPTIONS "--tool=memcheck")
SET (CTEST_TESTING_TIMEOUT 3600)
ctest_configure(BUILD "${CTEST_BINARY_DIRECTORY}" )
ctest_build(BUILD "${CTEST_BINARY_DIRECTORY}" )
ctest_test(BUILD "${CTEST_BINARY_DIRECTORY}" )
ctest_memcheck(BUILD "${CTEST_BINARY_DIRECTORY}" )
ctest_coverage(BUILD "${CTEST_BINARY_DIRECTORY}" )
ctest_submit()

1. Memory checking: This didn't work, CTest complained, that it could
not find the memory checker. From the source code I had the impression,
that valgrind should work
    automatically if found. The relevant output from ctest is the following:
SetCTestConfigurationFromCMakeVariable:MemoryCheckCommand:CTEST_MEMORYCHECK_COMMANDSetCTestConfiguration:MemoryCheckCommand:valgrind
SetCTestConfigurationFromCMakeVariable:MemoryCheckCommandOptions:CTEST_MEMORYCHECK_COMMAND_OPTIONSSetCTestConfiguration:MemoryCheckCommandOptions:--tool=memcheck
SetCTestConfiguration:BuildDirectory:/tmp/new_arch_nightly_build/obj/dbg
SetCTestConfiguration:SourceDirectory:/tmp/new_arch_nightly_build
Memory check project /tmp/new_arch_nightly_build/obj/dbg
Memory checker (MemoryCheckCommand) not set, or cannot find the
specified program.

2. Also the coverage command did not work, although the compile flags
have been set correctly (with -fprofile-arcs etc.). The .gcda files are
produced, which proves, that the
    binaries are build correctly, but it seems that gcov is not called.
I haven't found a possibility to explicitly set the command name of the
gcov binary, which is gcov-4.2 in my case.
    The relevant output here is this:

 Add coverage exclude regular expressions.
SetCTestConfiguration:BuildDirectory:/tmp/new_arch_nightly_build/obj/dbg
SetCTestConfiguration:SourceDirectory:/tmp/new_arch_nightly_build
Performing coverage
 COVFILE environment variable not found, not running  bullseye
   Processing coverage (each . represents one file):
    ."" -l -o
"/tmp/new_arch_nightly_build/obj/dbg/project/CMakeFiles/Project.dir"
"/tmp/new_arch_nightly_build/obj/dbg/project/CMakeFiles/Project.dir/ProjectElement.cpp.gcda"
No such file or directory
..... repeated for lots of source files
Cannot find any coverage files. Ignoring Coverage request.

3. I have one test, which takes longer than the default 10 minutes per
test, which has been killed. So I assume that CTEST_TESTING_TIMEOUT has
been ignored
    or is not the right variable to set.
4. I have searched for a possibility to exclude certain tests from
memory checking from within the script, but haven't found any.

So this is it. If you could help me to get this running, I would be very
grateful. Currently I don't have the time to search through the source
code to find these things out myself :-)

Regards,

Martin
____________
Virus checked by G DATA AntiVirus
Version: AVF 19.142 from 12.11.2008