[CMake] Announcing CMake BASIS, a set of CMake based project interoperability and automation tools

[Andreas Schuh]

On Jan 23, 2014, at 9:40 AM, Stephen Kelly <steveire at gmail.com> wrote:

> Andreas Schuh wrote:
>> How often have you seen CMake code as the following
>> 
>> add_executable(foo foo.cpp)
>> 
>> ?
> 
> I see executables with a single source file only in dummy test code, and 
> even then 
> 
> add_executable(foo main.cpp)
> 
> is more common.

I presume that’s because you mostly worked on larger, mainly C++ or other compiled language, based projects. At least in research software you will find these quite frequently, where there are dozens of executables implemented in single .cpp modules. These may just link to libraries which are build from a multiple .cpp modules (that is why obviously add_library would not support such short notation), but yet, the executable itself does (and IMHO should) only consist of a single .cpp module which defines the entry point (main). If you name all these main.cpp, you are not giving away what binary file this .cpp module relates to and further have to have them all in different subdirectories… Isn’t it nicer to have them all in one directory with meaningful names ? I believe program logic should go into libraries so it could be reused by other executables or packages if there need be.

> 
>> 
>> The basis_add_executable supports this use case as well, but additionally
>> the shorter notation (not counting the basis_ prefix ;) )
>> 
>> basis_add_executable(foo.cpp)
> 
> Yes, that's what I was referring to. I see why you have it. I'm not 
> convinced it should be upstreamed.

I guess most people can live with defining a CMake macro for it if they want to.

> 
>> is intended to bridge both, make the use of
>> globbing expressions convenient and safe at the same time.
> 
> I would like to see investigation into what can be done upstream to benefit 
> cmake users who do not use BASIS.
> 
>>> Another example: You have code for adding scripts as executables. What
>>> are the generic (non-BASIS related) use cases for that?
>>> 
>> You can define dependencies among them or to modules written in the same
>> scripting language. 
> 
> Can you say why this is useful? 
> 
> Is it only useful in cases where the buildsystem creates a 
> particular/modified script from an input, such as by replacing @VAR@? 
> 
>> You can have find_package calls to find installed
>> modules and declare that your script depends on them.
> 
> Why is this useful? If my script depends on something external I require the 
> find_package. Can you be more specific on what BASIS does in addition here?
> 
>> An installation rule
>> for the script may ensure that the modules get installed as well. 
> 
> I can imagine that defining, say, a python library as such for packaging and 
> installation purposes could be generically useful, but I'm not sure that 
> that is what you are saying. I'm trying to find descriptions of things in 
> BASIS that should be in cmake, that show gaps in cmake functionality etc.

Think of script modules the same way you think about shared libraries in case of binary code. If you have multiple versions of a library installed, you have to specify the LD_LIBRARY_PATH or presumably better set the RPATH of the executable properly. Most scripting languages have something similar as the LD_LIBRARY_PATH and it is possible to adjust the search path within the script. To automatically generate code that does the latter to avoid the problem of having to set the module search path right, especially considering users not familiar enough with these concepts and in cases where this is not trivial due to different scripts requiring different versions of a module and thus a change of the environment, you need the information about which modules a particular script depends on along with the imported locations of the used modules. This you can get with CMake via the import targets and the dependency graph among scripts and modules.

Not sure if you would like such functionality in a tool such as CMake, but for the purposes of BASIS it worked out well. At the top of each executable script, BASIS inserts the appropriate code to prepend the default “RPATH” of the scripts to the search path. Relative paths where appropriate to maintain relocate-abilitiy as much as possible, absolute paths otherwise. Yet, if an installation would be relocated, using the other means of setting the environment variables can possibly fix any potentially broken search paths. But that is also what you get with the RPATH of binary executables. Installations are generally not meant to be moved around. The higher priority is ensuring that scripts use the right version of the modules they were “linked” to.

> 
>> When
>> wrapping scripts in Windows Command files to mimic the she-bang directive
>> of Unix, calling such script becomes just as convenient as it is on Unix
>> from the command line without having to specify the interpreter executable
>> explicitly.
> 
> I'm not familiar with Windows, but this comment doesn't seem to relate to 
> cmake?

Maybe not to CMake, but whatever is taking care of the build process. Obviously you can achieve this using custom CMake macros/functions/commands.

In case of BASIS, it’s a custom build command which wraps the executable script into a Windows NT Script file (.cmd extension). Otherwise you need to preserve the file name extension so Windows knows via file associations which program to open this file with. I prefer, however, the same usage of the command-line tools of my software packages on Windows as I would on Unix. Then I can also life with only one usage documentation for both OS’s.

For example, in case of Python, you can “wrap” the script and have it act as both Windows NT Script and Python script by adding the following line at the top

@setlocal enableextensions & "/path/to/required/python/version" -x "%~f0" %* & goto :EOF
% Any Python code goes below this line

Python will ignore the first line. In case of Perl, however, you need to do a bit more:

@goto = "START_OF_BATCH” ;
@goto = ();
% Any Perl code goes in between these lines
__END__:
“START_OF_BATCH"
@“/path/to/required/perl" -w -S "%~f0" %*

As this modified Perl script is no longer valid Perl and because you need to replace the path to the found interpreter executables in both cases (also for Unix she-bang directives), you need to make this modification at build time. To not change the line numbers of error messages, BASIS recommends to have at least one blank line at the top of each script where it can insert any needed code. That is usually anyway the case after or before the header comment. For every extra line added at the top, it will remove one blank line if possible to keep the balance.

> 
>> 
>>> 
>>> Should CMake be extended with similar capabilities somehow?
>>> 
>>> I personally believe this could be very useful to quite a few users.
>>> 
>>> Would it help if an IMPORTED executable could be a script?
>>> 
>>> What would this entail? Is there perhaps another question behind this
>>> question?
> 
>> Instead of relying on the actual name of the installed
>> executable file, the build target is imported and any external executables
>> are only called by their assigned build target name. 
> 
> Yes.
> 
>>> 2) Do you generally recommend that CMake users use BASIS from now on, or
>>> is it something you expect will be niche/mostly used by yourselves?
>>> 
>> 
>> The first and foremost target group are clearly research groups 
> 
> Yes.
> 
>> and less
>> experienced software developers or those who do not want to bother writing
>> a considerable amount of CMake code rather than their actual software.
> 
> I'm not personally convinced your approach is the best one, but it's not 
> aimed at me anyway. I'm only interested in finding upstreamable features and 
> concepts, and finding gaps in the cmake offering.

We are open to any suggestions for improvements.

The purpose of BASIS is to slim down much of what you can do with CMake/CPack/CTest/CDash to what is needed by our target group without requiring people to learn every aspect of the yet ever changing “modern CMake” approach. Even writing a FindPackage.cmake module is not standardized as much as it could be. Defining such standard and offering functionality for the most common use cases is what BASIS does. While the “modern CMake” may change and needs retraining, BASIS can adapt and use new features without changing its user API (as long as possible). This may be not what professionals in a software development position prefer, where you may have one or two “CMake gurus” who take care of things or you may be one yourself, but in less software development oriented environments where people write software as a tool to express and experiment with new algorithms. For these fellows, whoever develops BASIS is supposed to know well about CMake and recent features, utilizing them where needed, and transferring their knowledge downstream transparent to the end user (e.g., research developer) via the CMake BASIS package.

> 
>>> 3) BASIS requires me to use basis_add_executable, KDE4 requires me to use
>>> kde4_add_executable, VTK requires me to use vtk_add_executable.
>>> 
> 
>> Initially I started off overriding the original CMake commands
> 
> Yeah, don't do that... :)

And in the end I have not. It seemed you suggested it though, but I may have mistaken your previous comment.

> 
>> On a side note, just today a co-worker asked me why the compiler cannot
>> find the header files when they were provided as additional arguments to
>> the add_executable command. Indeed this was a reasonable assumption, I
>> think. Why the need for an additional include_directories when I already
>> specified where the header files are located ? This is not something BASIS
>> is taking care of yet either, but would certainly be one of the things I
>> would consider adding.
> 
> This is the kind of concrete feedback *about cmake* that I'm looking for in 
> this discussion. It's self contained, refers to existing cmake 
> functionality, it's easy to imagine the 'user story' and what the user code 
> would look like etc.
> 
> I don't have any comment about the viability of adding such functionality to 
> cmake add_* commands though.
> 
> Thanks,
> 
> Steve.
> 
> 

Cheers,

Andreas

> 
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake