[CMake] CTest model deviates from the requsted one

[Matějů Miroslav, Ing.]

Hi,

I would like to report the following problem. I experienced that CTest running a CTestScript with a manually specified model (e.g. CTEST_START(Experimental)) sometimes deviates from the selected model. For example, when the test model Nightly is requested, the test sometimes gets reported to CDash as Continuous; the Experimental model might get reported as Nightly or Continuous. I started to experience this problem massively when I started to run Continuous tests in addition to Nightly tests on my server.

I tried to add debug messages to the CTest and shell scripts which control the testing process in my environment. I found out that the wrong decision happens during the CTEST_START call.

The code:

IF (NOT BUILD_MODEL)
	MESSAGE (FATAL_ERROR "Build model is not set")
ENDIF (NOT BUILD_MODEL)
CTEST_EMPTY_BINARY_DIRECTORY ("${CTEST_BINARY_DIRECTORY}")
MESSAGE(STATUS "Build model: ${BUILD_MODEL}")
CTEST_START (${BUILD_MODEL})
MESSAGE(STATUS "Build model: ${BUILD_MODEL}")

renders (using ctest -VV):

-- Build model: Experimental
SetCTestConfiguration:SourceDirectory:/home/ctest/trunk/tests/moduletests/cm
SetCTestConfiguration:BuildDirectory:/home/ctest/trunk/_build/moduletests/cm
Run dashboard with model Experimental
   Source directory: /home/ctest/trunk/tests/moduletests/cm
   Build directory: moduletests/cm
   Reading ctest configuration file: /home/ctest/trunk/tests/moduletests/cm/CTestConfig.cmake
SetCTestConfigurationFromCMakeVariable:NightlyStartTime:CTEST_NIGHTLY_START_TIME
SetCTestConfiguration:NightlyStartTime:01:00:00 UTC
SetCTestConfigurationFromCMakeVariable:Site:CTEST_SITE
SetCTestConfiguration:Site:MiM-testsrv-ubuntu
SetCTestConfigurationFromCMakeVariable:BuildName:CTEST_BUILD_NAME
SetCTestConfiguration:BuildName:cm
   Site: MiM-testsrv-ubuntu
   Build name: cm
   Use Continuous tag: 20170912-0800
-- Build model: Experimental

I tried to upgrade CMake from the distribution-provided version up to 3.9.1 but the problem persists.

I noticed that the problem appears after changing the model in the same build directory, despite it has been wiped using CTEST_EMPTY_BINARY_DIRECTORY. CTest seems to prefer the previously used model sometimes instead of the explicitly given one (using CTEST_START). So I am currently trying to work around it by using separate build directories for each model. Is it the correct way or would you recommend something else?

Best regards,

Ing. Miroslav Matějů
Programmer Analyst

AŽD Praha s.r.o.
Technology Division
Research and Development
Žirovnická 2/3146, 106 17  Prague
Czech Republic
Phone: +420 267 287 476
Web: www.azd.cz