[CMake] CTest with multiple directories: problem with absolute paths

[Tyler Roscoe]

Bill is a colleague of mine, and we are working together on this
problem.

On Fri, May 29, 2009 at 10:58:50AM -0700, Bill V wrote:
> The subdir code in CMAKE proper (cmSubdirCommand.cxx) has logic to
> handle relative or absolute paths.   The subdir handling code in
> CTest/cmCTestTestHandler.cxx  always assumes it's a relative path and
> prepends the current directory.     Maybe cmCTestTestHandler.cxx needs
> to be updated.

Indeed. After a lot of experimenting, I think I have a small case that
demonstrates the problem.

[tylermr at alta:~/ctest-paths-test2]$ find
.
./module1
./module1/CMakeLists.txt

./module1/module1subdir
./module1/module1subdir/CMakeLists.txt

./module2
./module2/CMakeLists.txt

Module1 is the "main" project. It does add_subdirectory() on module2
(which lives next to it on disk) and module1subdir (which lives
underneath module1 on disk).

The Lists are also simple:

MODULE1
-------
project(module1)
cmake_minimum_required(VERSION 2.6)

enable_testing()

set (PLATFORM "linux")
set (BUILD_TYPE "debug")

# The source path of the subdirectory is relative to the source path of
# this project. The binary path of the subdirectory is relative to the
# *binary* path of this project, hence the extra "../".
add_subdirectory(../module2 ../../module2/${PLATFORM}/${BUILD_TYPE})
add_subdirectory(module1subdir ../module1subdir/${PLATFORM}/${BUILD_TYPE})

MODULE1SUBDIR
-------------
project(module1subdir)
cmake_minimum_required(VERSION 2.6)

add_test (module1subdirtest1 "pwd")

MODULE2
-------
project(module2)
cmake_minimum_required(VERSION 2.6)

add_test (module2test1 "pwd")



So let's do an out-of-source build on this project:

[tylermr at alta:~/ctest-paths-test2/module1]$ mkdir build
[tylermr at alta:~/ctest-paths-test2/module1]$ cd build/
[tylermr at alta:~/ctest-paths-test2/module1/build]$ cmake ..
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Check for working C compiler: /usr/lib64/ccache/gcc
-- Check for working C compiler: /usr/lib64/ccache/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/lib64/ccache/c++
-- Check for working CXX compiler: /usr/lib64/ccache/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Configuring done
-- Generating done
-- Build files have been written to:
/alta/tylermr/ctest-paths-test2/module1/build


And let's inspect the CTestTestfile.cmake which is written out:

[tylermr at alta:~/ctest-paths-test2/module1/build]$ cat CTestTestfile.cmake
# CMake generated Testfile for
# Source directory: /alta/tylermr/ctest-paths-test2/module1
# Build directory: /alta/tylermr/ctest-paths-test2/module1/build
#
# This file replicates the SUBDIRS() and ADD_TEST() commands from the
# source
# tree CMakeLists.txt file, skipping any SUBDIRS() or ADD_TEST()
# commands
# that are excluded by CMake control structures, i.e. IF() commands.
SUBDIRS(/alta/tylermr/ctest-paths-test2/module2/linux/debug)
SUBDIRS(../module1subdir/linux/debug)


Note that module2 is written out with an absolute path while
module1subdir is written out with a relative path. CTest will only work
with the relative path because, as Bill noted, CTest *always* prepends
the current binary directory to the beginning of each path it finds in a
SUBDIRS() command. As a result, CTest can only find one of the test
cases in the project:

[tylermr at alta:~/ctest-paths-test2/module1/build]$ ctest
Start processing tests
Test project /alta/tylermr/ctest-paths-test2/module1/build
  1/  1 Testing module1subdirtest1               Passed

100% tests passed, 0 tests failed out of 1


Proof of the path prepending can be seen in this excerpt from strace:

[tylermr at alta:~/ctest-paths-test2/module1/build]$ strace ctest
execve("/usr/local/bin/ctest", ["ctest"], [/* 89 vars */]) = 0
[...]
getcwd("/alta/tylermr/ctest-paths-test2/module1/build", 2048) = 46
chdir("/alta/tylermr/ctest-paths-test2/module1/build") = 0
access("/alta/tylermr/ctest-paths-test2/module1/build//alta/tylermr/ctest-paths-test2/module2/linux/debug", R_OK) = -1 ENOENT (No such file or directory)
getcwd("/alta/tylermr/ctest-paths-test2/module1/build", 2048) = 46
chdir("/alta/tylermr/ctest-paths-test2/module1/build") = 0
access("/alta/tylermr/ctest-paths-test2/module1/build/../module1subdir/linux/debug", R_OK) = 0
[...]


I think the this inconsistency stems from some logic in
cmLocalGenerator::ConvertToRelativePath (which is called by
cmLocalGenerator::Convert which is called by
cmLocalGenerator::GenerateTestFiles which is the method that writes out
CTestTestfile.cmake). I haven't read the code too closely but these
comments seem enlightening:

  // Skip conversion if the path and local are not both in the source
  // or both in the binary tree.

and

  // If no part of the path is in common then return the full path.


I believe there is a bug in CTest's SUBDIRS() command. It should accept
absolute paths, like add_subdirectory() does. 

Question time!
--------------

Is there a reason SUBDIRS doesn't accept absolute paths? If we wrote a
patch so that it did, could this patch be accepted into CMake proper?

Is there something wrong with a CMakeLists that does add_subdirectory()
on directories that don't happen to be located underneath that
CMakeLists file? We do this all over the place (which I can't imagine is
an unusual practice), so are we simply unable to use CTest until this
bug is fixed?

Is there a workaround to this problem that doesn't involve patching
CMake code? We discussed, e.g., doing a pass after the initial cmake
step to modify paths in all the generated CTestTestfile, but that's
obviously ugly.



Btw, there is a discrepancy between how subdirs works in CTest vs how it
works in CMake. Compare CTest/cmCTestTestHandler::cmCTestSubdirCommand
with cmSubdirCommand::cmSubdirCommand. 

The one for CMake has some logic that checks if
Makefile->GetCurrentDirectory/directory_from_subdir_command points
somewhere useful. If it does, cmSubdirCommand adds the directory with
the prepended current directory. (If not, it returns
directory_from_subdir_command undecorated.)

cmCTestSubdirCommand doesn't have this logic; it always prepends
GetCurrentWorkingDirectory() with no tests to make sure the resulting
decorated directory exists on the filesystem.

Can we just add the logic from cmSubdirCommand to cmCTestSubdirCommand
and resolve our problems?


Sorry this is so long, but I'm hoping it is complete and useful for
CMake developers or for others running into a similar,
difficult-to-diagnose problem with CTest and path shenanigans.

Thanks,
tyler