[CMake] ctest ques.: capture driver errs

[dean.inglis at camris.ca]

Hi David,

thanks for your reply.

I have a couple more questions (for anyone) based on
CMake testing of VTK/Examples/Build/vtkMy,
since I seem to be generating image test errors with ctest.

In my root CMakeLists.txt, I added the following to do 
hopefully basic testing, without submitting results to 
a dashboard: 

#projectroot/CMakeLists.txt

MARK_AS_ADVANCED(BUILD_TESTING)
IF(BUILD_TESTING)
  ENABLE_TESTING()
ENDIF(BUILD_TESTING)

and in CMakeOptions.cmake:

#projectroot/CMakeOptions.cmake

MARK_AS_ADVANCED(BUILD_TESTING)
IF(BUILD_TESTING)
  IF(NOT VTKLOCAL_TESTING_TEMPORARY_DIR)
    MAKE_DIRECTORY(${PROJECT_BINARY_DIR}/Testing/Temporary)
    SET(VTKLOCAL_TESTING_TEMPORARY_DIR "${PROJECT_BINARY_DIR}/Testing/Temporary" CACHE INTERNAL "Temporary folder for tests")
  ENDIF(NOT VTKLOCAL_TESTING_TEMPORARY_DIR)
  ENABLE_TESTING()
  OPTION(VTK_USE_DISPLAY "Turn this option off and tests will not popup windows" ON)
  MARK_AS_ADVANCED(VTK_USE_DISPLAY)
  SET(CXX_TEST_PATH ${EXECUTABLE_OUTPUT_PATH})
ENDIF(BUILD_TESTING)

finally, in one of my source code subdirs, I have Testing/Cxx/CMakeLists.txt
along with the .cxx tests:

#projectroot/Common/Testing/Cxx/CMakeLists.txt

SET(KIT LocalCommon)

SET(MyTests
  TestCommon.cxx
  TestCommonGeomUtils.cxx
  TestCommonGeomUtils2.cxx
)

CREATE_TEST_SOURCELIST(
  Tests
  ${KIT}CxxTests.cxx
  ${MyTests}
  EXTRA_INCLUDE vtkTestDriver.h
)

ADD_EXECUTABLE(${KIT}CxxTests ${Tests} )
TARGET_LINK_LIBRARIES(${KIT}CxxTests vtkLocalCommon vtkCommon vtkFiltering vtkGraphics vtkRendering)

SET (TestsToRun ${Tests})
REMOVE (TestsToRun ${KIT}CxxTests.cxx)

FOREACH (test ${TestsToRun})
  GET_FILENAME_COMPONENT(TName ${test} NAME_WE)
  ADD_TEST(
    ${TName} 
    ${CXX_TEST_PATH}/${KIT}CxxTests
    ${TName}
    -V ${VTKLOCAL_SOURCE_DIR}/Testing/baseline/${TName}.png
    -T ${VTKLOCAL_TESTING_TEMPORARY_DIR}
  )   
ENDFOREACH (test) 


The third .cxx test uses the standard vtk testing tools

<snip>
  // some VTK graphics pipeline here...

  int retVal = vtkRegressionTestImage( renWin );
  if ( retVal == vtkRegressionTester::DO_INTERACTOR)
	{
	iren->Start();
	}
  
  return retVal;
<snip>

running ctest with NO image in the project source tree's Testing/baseline dir generates
a (correct) .png test image in the project binary build tree under /Testing/Temporary,
with LastTest.log file containing:

<snip>
3/3 Testing: TestCommonGeomUtils2
3/3 Test: TestCommonGeomUtils2
Command: "d:/Developer/Releases/Borland/vtkStatic/bin/LocalCommonCxxTests.exe" "TestCommonGeomUtils2" "-V" "d:/Developer/ISSCode/vtkLocal/Testing/baseline/TestCommonGeomUtils2.png" "-T" "d:/Developer/Releases/Borland/vtkLocalStatic/Testing/Temporary" 
Directory: d:/Developer/Releases/Borland/vtkLocalStatic/Common/Testing/Cxx
"TestCommonGeomUtils2" start time: Sep 20 11:45 Eastern Daylight Time
Output:
----------------------------------------------------------
<DartMeasurement name="ImageNotFound" type="text/string">d:/Developer/ISSCode/vtkLocal/Testing/baseline/TestCommonGeomUtils2.png</DartMeasurement>

<DartMeasurement name="WallTime" type="numeric/double">0.0900002</DartMeasurement>

<DartMeasurement name="CPUTime" type="numeric/double">0.1</DartMeasurement>

----------------------------------------------------------
"TestCommonGeomUtils2" end time: Sep 20 11:45 Eastern Daylight Time
"TestCommonGeomUtils2" time elapsed: 00:00:00
----------------------------------------------------------
<snip>


If I copy that image as my baseline for the test into the source tree's Testing/baseline dir
and then run the test, I encounter a failed test with log entry along with a LastTestsFailed.log
and no .diff.png or .small.png image(s):

<snip>
3/3 Testing: TestCommonGeomUtils2
3/3 Test: TestCommonGeomUtils2
Command: "d:/Developer/Releases/Borland/vtkStatic/bin/LocalCommonCxxTests.exe" "TestCommonGeomUtils2" "-V" "d:/Developer/ISSCode/vtkLocal/Testing/baseline/TestCommonGeomUtils2.png" "-T" "d:/Developer/Releases/Borland/vtkLocalStatic/Testing/Temporary" 
Directory: d:/Developer/Releases/Borland/vtkLocalStatic/Common/Testing/Cxx
"TestCommonGeomUtils2" start time: Sep 20 11:48 Eastern Daylight Time
Output:
----------------------------------------------------------
<DartMeasurement name="ImageError" type="numeric/double">0</DartMeasurement><DartMeasurement name="BaselineImage" type="text/string">Standard</DartMeasurement><DartMeasurement name="WallTime" type="numeric/double">0.14</DartMeasurement>

<DartMeasurement name="CPUTime" type="numeric/double">0.14</DartMeasurement>

----------------------------------------------------------
"TestCommonGeomUtils2" end time: Sep 20 11:48 Eastern Daylight Time
"TestCommonGeomUtils2" time elapsed: 00:00:00
----------------------------------------------------------
<snip>

I thought at least that ctest would generate the usual .small and .diff images
in a failure case.  Also, I don't see why the test should fail since the test
image and baseline images are identical.

Dean