[CMake] confused by ctest
David Cole
david.cole at kitware.com
Tue Feb 3 15:41:31 EST 2009
Don't worry... it's not submitting results to Kitware, it's using a default
value for CTEST_TRIGGER_SITE, which is old and leftover from pre-CDash
days... It used to be the case that after all results were uploaded, a
"trigger" script would run to tell the server to process the previously
uploaded files. CDash processes files as they are submitted so triggers are
not necessary with CDash. I actually fixed this very code in ctest earlier
today to avoid trigger calls when using CDash. (See changes listed here:
http://www.cdash.org/CDash/viewUpdate.php?buildid=263494)
To answer your questions:
The CMake book has a Tutorial chapter in it that takes you through the steps
involved in adding CTest / CDash support to your CMake based project. If you
have the book, read through the Tutorial chapter... it's pretty quick. If
you don't have the book, buy it. :-) (You can also check out the code that
goes with the chapter in the CMake/Tests/Tutorial/Step1 through Step7
directories in a CMake source tree...)
Basically, you need to call ENABLE_TESTING() and INCLUDE(CTest) in your
CMakeLists.txt file.
And yes, "make test" returns errors when there are test failures...
And you should set CTEST_UPDATE_COMMAND to the full path to the svn
executable to use svn instead of cvs...
HTH,
David
On Tue, Feb 3, 2009 at 3:13 PM, Bill O'Hara <billtohara at gmail.com> wrote:
> Hello,
>
> I wonder if anyone can point me in the direction of a hello world project
> that successfully demonstrates the basic features of cmake+ctest+cdash? I'm
> using cmake/ctest version 2.6-patch 2 and cdash 1.2.1. Or somewhere with a
> working tutorial I can follow along? (I'm a bit confused by where to find
> the up to date documentation I should work from).
>
> I'm tasked with putting together a new build system for a commercial
> project but when evaluating cmake I can't seem to get the basics working
> even on trivial hello world projects. The comedy of errors I've managed so
> far is included below - I'd be extremely grateful for any hints. I'm
> particularly worried by the testing submitting results to kitware.com even
> though I specifically said we're using cdash and pointed it at my local
> cdash server (where some results are submitted despite other errors).
>
> Thanks in advance for any hints! I have marked QUESTION below at various
> points if anyone can spot my problems :(
>
> Bill
>
>
> #
> # Start from a clean setup.
> #
> bill-box:/tmp> ls svn
> README.txt conf dav db format hooks locks
> bill-box:/tmp> ls build/
> bill-box:/tmp> ls binary/
> bill-box:/tmp> ls source/
>
>
> #
> # Checkout has a minimal test project checked out from /tmp/svn:
> # - A shared library called impl build from include/header.h and
> impl/impl.c
> # - A test called foo that invokes the function from impl and always
> succeeds.
> # - A test called bar that invokes the function from impl and always fails.
> #
> bill-box:/tmp> ls checkout/
> CMakeLists.txt CTestConfig.cmake impl include tests
>
> #
> # CMakeLists.txt
> #
> bill-box:/tmp/checkout> cat CMakeLists.txt
> PROJECT (HELLOWORLD)
> CMAKE_MINIMUM_REQUIRED(VERSION 2.6)
> INCLUDE_DIRECTORIES (${CMAKE_SOURCE_DIR}/include)
> ADD_SUBDIRECTORY (${CMAKE_SOURCE_DIR}/impl)
> INCLUDE (CTest)
> ADD_SUBDIRECTORY (${CMAKE_SOURCE_DIR}/tests)
>
>
> #
> # CTestConfig.cmake
> #
> bill-box:/tmp/checkout> cat CTestConfig.cmake
> SET (CTEST_DROP_SITE_CDASH TRUE)
> SET (CTEST_PROJECT_NAME "helloworld")
> SET (CTEST_NIGHTLY_START_TIME "00:00:00 EST")
> SET (UPDATE_TYPE "true")
> SET (CTEST_DROP_METHOD "http")
> SET (CTEST_DROP_SITE "bill-box")
> SET (CTEST_DROP_LOCATION "/submit.php?project=helloworld")
>
>
> #
> # And an experimental Nightly ctest script to try to upload
> # test results to CDash after checking out and building
> # the project source code. This script is not in /tmp/checkout.
> #
> bill-box:/tmp> cat experimental.cmake
> SET (CTEST_DROP_SITE_CDASH TRUE)
> SET (CTEST_PROJECT_NAME "helloworld")
> SET (CTEST_NIGHTLY_START_TIME "00:00:00 EST")
> SET (UPDATE_TYPE "true")
> SET (CTEST_DROP_METHOD "http")
> SET (CTEST_DROP_SITE "bill-box")
> SET (CTEST_DROP_LOCATION "/submit.php?project=helloworld")
> SET (CTEST_SOURCE_DIRECTORY "/tmp/source")
> SET (CTEST_BINARY_DIRECTORY "/tmp/binary")
> SET (CTEST_START_WITH_EMPTY_BINARY_DIRECTORY TRUE)
> SET (CTEST_SVN_CHECKOUT "svn co file:///tmp/svn
> \"${CTEST_SOURCE_DIRECTORY}\"")
> SET (CTEST_CMAKE_GENERATOR "Unix Makefiles")
> SET (CTEST_CUSTOM_PRE_TEST "${CTEST_SVN_CHECKOUT}")
> CTEST_START(Nightly)
> CTEST_UPDATE(SOURCE "${CTEST_SOURCE_DIRECTORY}")
> CTEST_CONFIGURE(BUILD "${CTEST_BINARY_DIRECTORY}")
> CTEST_BUILD(BUILD "${CTEST_BINARY_DIRECTORY}")
> CTEST_TEST(BUILD "${CTEST_BINARY_DIRECTORY}")
> CTEST_SUBMIT()
>
>
> #
> # First, try to configure using cmake.
> #
> bill-box:/tmp> cd build
> bill-box:/tmp/build> cmake /tmp/checkout
> -- The C compiler identification is GNU
> -- The CXX compiler identification is GNU
> -- Check for working C compiler: /usr/bin/gcc
> -- Check for working C compiler: /usr/bin/gcc -- works
> -- Detecting C compiler ABI info
> -- Detecting C compiler ABI info - done
> -- Check for working CXX compiler: /usr/bin/c++
> -- Check for working CXX compiler: /usr/bin/c++ -- works
> -- Detecting CXX compiler ABI info
> -- Detecting CXX compiler ABI info - done
> -- Configuring done
> -- Generating done
> -- Build files have been written to: /tmp/build
>
>
> #
> # Now, build and test.
> #
> bill-box:/tmp/build> make all test
> Scanning dependencies of target impl
> [ 33%] Building C object impl/CMakeFiles/impl.dir/impl.c.o
> Linking C shared library libimpl.so
> [ 33%] Built target impl
> Scanning dependencies of target bar
> [ 66%] Building C object tests/CMakeFiles/bar.dir/bar.c.o
> Linking C executable bar
> [ 66%] Built target bar
> Scanning dependencies of target foo
> [100%] Building C object tests/CMakeFiles/foo.dir/foo.c.o
> Linking C executable foo
> [100%] Built target foo
> Running tests...
> Start processing tests
> Test project /tmp/build
> 1/ 2 Testing foo Passed
> 2/ 2 Testing bar ***Failed
>
> 50% tests passed, 1 tests failed out of 2
>
> The following tests FAILED:
> 2 - bar (Failed)
> Errors while running CTest
> make: *** [test] Error 8
>
>
> # QUESTION
> #
> # Is this Error 8 from make the expected result after some tests have
> failed?
> #
>
>
> #
> # Now try the Nightly test target.
> #
> bill-box:/tmp/build> make Nightly
> Scanning dependencies of target Nightly
> Site: bill-box
> Build name: Linux-c++
> Determine Nightly Start Time
> Specified time: 00:00:00 EST
> Create new tag: 20090203-0500 - Nightly
> Start processing tests
>
> #
> # QUESTION
> # Why does it think we are using CVS here?
> #
>
> Updating the repository
> Updating the repository: /tmp/checkout
> Use CVS repository type
> Determine Nightly Start Time
> Specified time: 00:00:00 EST
> Gathering version information (each . represents one updated file):
> Project is up-to-date
> Update with command: "/usr/bin/cvs" -z3 update -d -A -P -D "2009-02-03
> 05:00:00 UTC" failed
> Configure project
> Each . represents 1024 bytes of output
> . Size of output: 0K
> Build project
> Each symbol represents 1024 bytes of output.
> '!' represents an error and '*' a warning.
> . Size of output: 0K
> 0 Compiler errors
> 0 Compiler warnings
> Test project /tmp/build
> 1/ 2 Testing foo Passed
> 2/ 2 Testing bar ***Failed
>
> 50% tests passed, 1 tests failed out of 2
>
> The following tests FAILED:
> 2 - bar (Failed)
>
> #
> # QUESTION
> # How do I tell it to not do Coverage?
> #
>
> Performing coverage
> Cannot find any coverage files. Ignoring Coverage request.
> Submit files (using http)
> Using HTTP submit method
> Drop site: http://bill-box/submit.php?project=helloworld
> Uploaded: /tmp/build/Testing/20090203-0500/Build.xml
> Uploaded: /tmp/build/Testing/20090203-0500/Configure.xml
> Uploaded: /tmp/build/Testing/20090203-0500/Test.xml
> Uploaded: /tmp/build/Testing/20090203-0500/Update.xml
> Submission successful
> Errors while running CTest
> make[3]: *** [CMakeFiles/Nightly] Error 8
> make[2]: *** [CMakeFiles/Nightly.dir/all] Error 2
> make[1]: *** [CMakeFiles/Nightly.dir/rule] Error 2
> make: *** [Nightly] Error 2
>
> #
> # QUESTION
> # Why are we getting 3 different Errors from make?
> #
>
>
> #
> #
> # Let us try that again using ctest with verbosity turned on.
> #
> #
>
> bill-box:/tmp/build> ctest -VV
>
> #
> # QUESTION
> # Where is this DartConfiguration coming from? We're trying
> # to use CDash.
> #
>
> UpdateCTestConfiguration from :/tmp/build/DartConfiguration.tcl
> Parse Config file:/tmp/build/DartConfiguration.tcl
> Start processing tests
> UpdateCTestConfiguration from :/tmp/build/DartConfiguration.tcl
> Parse Config file:/tmp/build/DartConfiguration.tcl
> Test project /tmp/build
> Constructing a list of tests
> Done constructing a list of tests
> Changing directory into /tmp/build/tests
> 1/ 2 Testing foo
> Test command: /tmp/build/tests/foo foo.c
> Test timeout computed to be: 1500
> 42
> -- Process completed
> Passed
> 2/ 2 Testing bar
> Test command: /tmp/build/tests/bar bar.c
> Test timeout computed to be: 1500
> 42
> -- Process completed
> ***Failed
>
> 50% tests passed, 1 tests failed out of 2
>
> The following tests FAILED:
> 2 - bar (Failed)
> Errors while running CTest
>
> #
> #
> # Try our experimental Nightly test script to check out and test the
> project.
> #
> #
>
> bill-box:/tmp> ctest -S experimental.cmake
>
> #
> # Drat; its complaining about the source control system and trying
> # to build and update before running my svn checkout successfully.
> #
> # It also fails to find my tests.
> #
>
> Cannot find CVSCommand, SVNCommand, or UpdateCommand key in the
> DartConfiguration.tcl
> Error(s) when updating the project
> Error(s) when building project
> A /tmp/source/impl
> A /tmp/source/impl/impl.c
> A /tmp/source/impl/CMakeLists.txt
> A /tmp/source/tests
> A /tmp/source/tests/foo.c
> A /tmp/source/tests/bar.c
> A /tmp/source/tests/CMakeLists.txt
> A /tmp/source/include
> A /tmp/source/include/header.h
> A /tmp/source/CTestConfig.cmake
> A /tmp/source/experimental.cmake
> A /tmp/source/CMakeLists.txt
> Checked out revision 9.
> No tests were found!!!
>
> #
> # QUESTION
> # Why were my tests not found?
>
> #
> #
> # Try that again with verbosity; the previous attempt seems to have
> # helped by eventually checking out source code.
> #
> #
>
> bill-box:/tmp> ctest -VV -S experimental.cmake
> * Extra verbosity turned on
> Reading Script: /tmp/experimental.cmake
> SetCTestConfiguration:SourceDirectory:/tmp/source
> SetCTestConfiguration:BuildDirectory:/tmp/binary
> Run dashboard with model Nightly
> Source directory: /tmp/source
> Build directory: /tmp/binary
> Reading ctest configuration file: /tmp/source/CTestConfig.cmake
> SetCTestConfigurationFromCMakeVariable:NightlyStartTime:CTEST_NIGHTLY_START_TIMESetCTestConfiguration:NightlyStartTime:00:00:00
> EST
> Site:
> Build name:
> Determine Nightly Start Time
> Specified time: 00:00:00 EST
> Use Nightly tag: 20090203-0500
> SetCTestConfiguration:SourceDirectory:/tmp/source
> SetCTestConfiguration:BuildDirectory:/tmp/binary
> SetCTestConfiguration:SourceDirectory:/tmp/source
> Updating the repository
> Updating the repository: /tmp/source
> Cannot find CVSCommand, SVNCommand, or UpdateCommand key in the
> DartConfiguration.tcl
> SetCTestConfiguration:BuildDirectory:/tmp/binary
> SetCTestConfiguration:SourceDirectory:/tmp/source
> SetCTestConfiguration:ConfigureCommand:"/opt/cmake-2.6.2/bin/cmake" "-GUnix
> Makefiles" "/tmp/source"
> SetCTestConfiguration:BuildDirectory:/tmp/binary
> SetCTestConfiguration:SourceDirectory:/tmp/source
> Configure project
> Configure with command: "/opt/cmake-2.6.2/bin/cmake" "-GUnix Makefiles"
> "/tmp/source"
> Run command: "/opt/cmake-2.6.2/bin/cmake" "-GUnix Makefiles" "/tmp/source"
> Each . represents 1024 bytes of output
> .-- Configuring done
> -- Generating done
> -- Build files have been written to: /tmp/binary
> Size of output: 0K
> Command exited with the value: 0
> SetMakeCommand:/usr/bin/gmake -i
> SetCTestConfiguration:MakeCommand:/usr/bin/gmake -i
> SetCTestConfiguration:BuildDirectory:/tmp/binary
> SetCTestConfiguration:SourceDirectory:/tmp/source
> Build project
> MakeCommand:/usr/bin/gmake -i
> Run command: "/usr/bin/gmake" "-i"
> Each symbol represents 1024 bytes of output.
> '!' represents an error and '*' a warning.
> .[ 33%] Built target impl
> [ 66%] Built target bar
> [100%] Built target foo
> Size of output: 0K
> Command exited with the value: 0
> 0 Compiler errors
> 0 Compiler warnings
> SetCTestConfiguration:BuildDirectory:/tmp/binary
> SetCTestConfiguration:SourceDirectory:/tmp/source
> Test project /tmp/binary
> Run command: svn co file:///tmp/svn "/tmp/source"
> Checked out revision 9.
> Constructing a list of tests
> Changing directory into /tmp/binary/tests
> 1/ 2 Testing foo
> Test command: /tmp/binary/tests/foo foo.c
> Test timeout computed to be: 600
> 42
> -- Process completed
> Passed
> 2/ 2 Testing bar
> Test command: /tmp/binary/tests/bar bar.c
> Test timeout computed to be: 600
> 42
> -- Process completed
> ***Failed
>
> 50% tests passed, 1 tests failed out of 2
>
> The following tests FAILED:
> 2 - bar (Failed)
> * Use default trigger site:
> http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.cgi
> SetCTestConfiguration:DropMethod:http
> SetCTestConfiguration:DropSite:bill-box
> SetCTestConfiguration:DropLocation:/submit.php?project=helloworld
> SetCTestConfiguration:TriggerSite:
> http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.cgi
> SetCTestConfiguration:BuildDirectory:/tmp/binary
> SetCTestConfiguration:SourceDirectory:/tmp/source
> Submit files (using http)
> Using HTTP submit method
> Drop site: http://bill-box/submit.php?project=helloworld
> Upload file: /tmp/binary/Testing/20090203-0500/Build.xml to
> http://bill-box/submit.php?project=helloworld&FileName=______20090203-0500-Nightly___XML___Build.xmlSize: 909
> Uploaded: /tmp/binary/Testing/20090203-0500/Build.xml
> Upload file: /tmp/binary/Testing/20090203-0500/Configure.xml to
> http://bill-box/submit.php?project=helloworld&FileName=______20090203-0500-Nightly___XML___Configure.xmlSize: 1066
> Uploaded: /tmp/binary/Testing/20090203-0500/Configure.xml
> Upload file: /tmp/binary/Testing/20090203-0500/Test.xml to
> http://bill-box/submit.php?project=helloworld&FileName=______20090203-0500-Nightly___XML___Test.xmlSize: 2272
> Uploaded: /tmp/binary/Testing/20090203-0500/Test.xml
> #
> # QUESTION
> # What the...?
> #
> # Why is it submitting results to kitware.com?!
> #
> #
> #
> Using HTTP trigger method
> Trigger site:
> http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.cgi
> Trigger url:
> http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.cgi?xmlfile=______20090203-0500-Nightly___XML___Build.xml
>
> Trigger url:
> http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.cgi?xmlfile=______20090203-0500-Nightly___XML___Configure.xml
>
> Trigger url:
> http://public.kitware.com/cgi-bin/Submit-Random-TestingResults.cgi?xmlfile=______20090203-0500-Nightly___XML___Test.xml
>
> Dart server triggered...
> Submission successful
> bill-box:/tmp>
>
>
>
>
> _______________________________________________
> CMake mailing list
> CMake at cmake.org
> http://www.cmake.org/mailman/listinfo/cmake
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20090203/7dea87fe/attachment-0001.htm>
More information about the CMake
mailing list