[CMake] CMake performs search for includes/libs in non-default compiler search paths.

[Michael Hertling]

On 01/28/2011 06:44 AM, Michael Wild wrote:
> On 01/27/2011 08:08 PM, Óscar Fuentes wrote:
>> Michael Wild <themiwi at gmail.com> writes:
>>
>>>> Okay, cmake removes duplicated directories that already are on the
>>>> list. So what?
>>>>
>>>> First, I hope that cmake does not optimize this series:
>>>>
>>>> A B C A
>>>>
>>>> ... to this:
>>>>
>>>> A B C
>>>>
>>>> or to this:
>>>>
>>>> B C A
>>>>
>>>> Removing adjacent duplicates is fine, but removing duplicates in general
>>>> makes difficult to debug the build and may affect semantics.
>>>>
>>>> Second, the topic here is that we are forced to jump through hops
>>>> because cmake tries to be helpful. It looks for files into places but
>>>> then it is up to us to figure out if those places must be added to the
>>>> list of header and library paths. That's not convenient at all, quite
>>>> the contrary, it is messy and confusing, as we are experiencing.
>>>
>>> The point I (and Andreas) wanted to make is that you can just add it. If
>>> it's a system directory, CMake is going to filter it out (I didn't say
>>> duplicates, I said "doesn't even show up ONCE", which means *zero* times).
>>
>> This is a moot point, because /usr/local/include and several other paths
>> which are searched by default by cmake are not system directories in
>> FreeBSD.
> 
> No it's not. If you have e.g. /usr/local/include/foo.h and you do
> 
> find_path(FOO_INCLUDE_DIR foo.h)
> if(FOO_INCLUDE_DIR)
>   include_directories("${FOO_INCLUDE_DIR}")
> endif()
> 
> you don't have to care, because -I/usr/local/include will show up on the
> command line. If foo.h happened to be in /usr/include, things would be
> (kind of) fine again, because CMake filters that one out and thus
> doesn't clutter your command line.
> 
> After all, you do use CMake because it's smarter than your compiler.
> Otherwise you could just use some dump shell script.
> 
>>
>>> And yes, CMake simplifies A B C A to A B C, which is perfectly fine
>>> because the trailing A doesn't have any effect whatsoever.
>>
>> The semantics of #include is one of the most underspecified points of
>> the C++ standard. It is up to the compiler vendor to decide what to do
>> with #include. Each vendor uses its own algorithm (unless they decided
>> on cloning the behavior of somebody else's compiler) and I've suffered
>> subtle bugs due to that fact while porting applications. IIRC the
>> standard says nothing about rewinding the search list for every lookup,
>> but I'm ready to admit that in practice removing trailing duplicates
>> makes no effect for all existing compilers and the vast majority of
>> cmake users will appreciate that feature, although I wouldn't bet my
>> neck on it.
> 
> That's true. But I bet that no vendor would be that insane to search the
> include-directories in reverse order, or make it probabilistic, making
> it more likely that a header gets picked up from a certain directory if
> it is mentioned more often ;-)
> 
>>
>>> But you could have found that out by yourself in about 20 seconds...
>>
>> Please spare the inflammatory language for some other thread. Thanks.
> 
> Sorry for being snarky. I can only blame a foul mood due to a nasty cold
> I caught...
> 
>>
>>> IMHO you're the one trying to jump through hoops that aren't even there...
>>
>> Okay, so if I have this:
>>
>> check_include_file(foo.h HAVE_FOO_H)
>>
>> and it succeeds, I have no guarantees that a program such as this:
>>
>> #include "foo.h"
>>
>> will compile, becasue foo.h may be found on a directory not included on
>> the compiler's default search list.
>>
>> Okay, so I have to add something like
>>
>>   figure_out_where_foo_h_really_is
>>   include_directories( where_foo_h_is )
>>
>> Not nice, but let's go ahead.
>>
>> We end having something like /usr/local/include (or /opt/local/include
>> or /opt/pkg/include etc) on the search list.
>>
>> We now test for the presence of some third party library ("thelib"). We
>> have an user-settable variable for using just in case we want to use an
>> specific install, and we use it:
>>
>> cmake -DTHELIB_IS_HERE=/home/oscar/thelib .
>>
>> The search mechanism for thelib looks into that directory first and
>> succeeds, as it should. We add the corresponding directory:
>>
>> include_directories( ${THELIB_INCLUDE_PATH} )
>>
>> But there is another install of thelib on /usr/local, and that install
>> is the one that will be used by our build because we end having this
>> search list:
>>
>> /usr/local/include /home/oscar/thelib/include
>>
>> Have I missed something?
> 
> I reiterate Andreas' answer from yesterday. But there IS one point where
> the removal of system directories could wreak potential havoc, but even
> then, there's nothing CMake could do about it. Assume this situation:
> 
> /usr/include/foo.h
> /usr/local/include/foo.h
> /usr/local/include/bar.h
> 
> Now, if you want to mix-and-match, by having this include-path:
> -I/usr/include -I/usr/local/include
> 
> (i.e. use /usr/include/foo.h and /usr/local/include/bar.h) you're in
> trouble, because CMake will filter out the /usr/include directory,
> leaving you with /usr/local/include/foo.h being found. Rats.
> 
> However, even if CMake didn't filter out, this situation could easily
> break, because the author of the CMakeLists.txt file usually isn't aware
> of your special situation. So he writes something like this:
> 
> find_path(BAR_INCLUDE_DIR bar.h)
> find_path(FOO_INCLUDE_DIR foo.h)
> include_directories(${BAR_INCLUDE_DIR} ${FOO_INCLUDE_DIR})
> 
> This results in (assuming CMake didn't filter):
> 
> -I/usr/local/include -I/usr/include
> 
> BOOOM. Reversing the directories in the include_directories() call would
> solve the problem in this particular situation, but might break on
> another machine, where the bar.h file is present twice. The problem is
> always there if you have multiple packages installed in the same prefix
> and you have multiple prefixes with containing conflicting packages.
> There's really nothing CMake can do about this, it's an intrinsic flaw
> of how includes are handled.

Just some remarks from my side:

At first, Óscar's objections that a liberal use of INCLUDE_DIRECTORIES()
may lure the preprocessor into the wrong directories when seraching for
headers cannot be denied. As an example - similar to Michael's above but
without the specially treated /usr/include - suppose two packages A and
B installed in /usr/local and a private version of B installed in $HOME.
IMO, this is not uncommon. A CMakeLists.txt wants to use {A,B} and says:

FIND_PATH(A_INCLUDE_DIR A.h)
FIND_PATH(B_INCLUDE_DIR B.h)
INCLUDE_DIRECTORIES(${A_INCLUDE_DIR} ${B_INCLUDE_DIR})

If you configure with CMAKE_PREFIX_PATH=$HOME, e.g., CMake would find
$HOME/include/B.h, but the preprocessor will use /usr/local/include/B.h
since the command line contains /usr/local/include before $HOME/include.
Changing the order of INCLUDE_DIRECTORIES()'s arguments, as Andreas has
pointed out, would fix this, but swapping the roles of A and B makes it
break again, and usually, the author of a CMakeLists.txt can't know the
right order - if there is any - a priori.

As a possible workaround, one might set CMAKE_C_FLAGS=-I$HOME/include
as this precedes INCLUDE_DIRECTORIES()'s arguments and is not subject
to CMake's optimization of the include directories' arrangement on the
command line, so one can even place /usr/include explicitly before any
other path.

However, there're configurations which can't be solved in such a way at
all: Suppose {A,B} in /usr/local and private versions of them in $HOME.
How to compile a source file that is to include /usr/local/include/A.h
and $HOME/include/B.h? AFAICS, this is simply not possible, but it is
nothing CMake can be blamed for in any manner.

As Michael has stated correctly, problems of that kind arise when there
are multiple installations of the same package and one of them resides
along with other packages under the same prefix. So, here lies the last
resort to address the issue: If the CMake/environment variables as well
as a possibly tweaked CMakeLists.txt keep failing, at least one of the
concerned packages needs to be installed somewhere else, i.e. under a
different prefix - perhaps additionally, perhaps one can get off with
some symlinks, but it must be solved by administrative means - not by
configurational ones.

Furthermore, leaving out FIND_PATH()'s result in INCLUDE_DIRECTORIES()
tends to cause trouble, too, as the OP's report clearly shows, but the
fact that CMake looks in more directories than the preprocessor is not
the root of these problems. Rather - to reiterate Michael again - it's
the manner the preprocessor searches include directories for headers,
and that's something beyond CMake's scope. BTW, the same difficulties
appear with libraries if one doesn't use full paths, i.e. using -L/-l
instead. Here, it's also possible that the linker will take a library
different from the one CMake has found before. This is the reason why
INCLUDE_DIRECTORIES()'s counterpart, LINK_DIRECTORIES(), shouldn't be
used anymore. Unfortunately, one cannot specify full paths for header
files on the command line as it is strongly recommended for libraries.

In summary, my opinion on this issue is as follows:

- Do use the results of FIND_PATH() in INCLUDE_DIRECTORIES(), and use
  them in an order resonable from the CMakeLists.txt's point of view.
- Be aware of the preprocessor possibly finding wrong headers if
  there are multiply installed packages and common prefixes.
- If the preprocessor takes a wrong header, try to face this with
  CMAKE_PREFIX_PATH, CMAKE_C_FLAGS etc. or tweak the CMakeLists.txt.
- If those means fail or are inappropriate or inconvenient, do it the
  hard way: Separate the concerned packages, either by an additional
  installation or feigned by symlinks, and retry with the previous
  step although this approach is doubtlessly annoying.

Perhaps, one can consider to include special placeholder variables in
INCLUDE_DIRECTORIES(), e.g. EXTRA_INCLUDE_DIRS or target/directory-
specific like:

INCLUDE_DIRECTORIES(${<TARGET>_EXTRA_INCLUDE_DIRS} ...)
INCLUDE_DIRECTORIES(${<DIRECTORY>_EXTRA_INCLUDE_DIRS} ...)

These placeholders can be set on the command line or the GUI to hook
into the preprocessor's overall search path, but of course, such an
effort would need to be made systematically throughout the project.

Regards,

Michael