[CMake] Announcing CMake BASIS, a set of CMake based project interoperability and automation tools

[Andreas Schuh]

Hi Steve and all,
 
On Jan 23, 2014, at 12:47 AM, Andrew Hundt <athundt at gmail.com> wrote:

> 
> On Wed, Jan 22, 2014 at 5:44 PM, Stephen Kelly <steveire at gmail.com> wrote:
> Andrew Hundt wrote:
> 
> > CMake BASIS is a set of utilities and standards created with the goal of
> > making CMake projects and libraries very easy to create, share, and reuse.
> 
> Hello,
> 
> There seems to be several years of development behind this already. Is it
> newly public, or newly publicized? I've never heard of it before.
> 
> The about page and the 3.0 release on github explains the history reasonably well. Previously it was public but not publicized. For the v3.0 release we moved it to github, redesigned and rewrote much of the documentation, added custom project templates, and made everything else configurable. For example we needed take out previously hard coded items such as University of Pennsylvania logo (I'm at Carnegie Mellon).

I’ve been developing BASIS mainly to establish some software development and distribution practices in the research lab I was working and to reduce the repetition of CMake code across (the various small) projects. It has been public, but I mainly used within this lab. Since then, however, I changed position and took the project with me with the hope to make it more generally useful and to offer it as a resource for various CMake and other auxiliary functions.

The main goal of BASIS was to set a certain “standard” for project organization and everything related to it for researchers who are not very familiar with common practices and especially not with the many possibilities and features of the whole CMake suite. Further we wanted to unify the look and feel of research software developed by individuals and yet distributed as open source by one lab, namely SBIA at Penn. It has been thus far only used by myself to repackage the research software that was developed at this lab while I was employed there. But it clearly has potential to be extended and used also by more experienced developers.

>  
> 
> > Website: http://opensource.andreasschuh.com/cmake-basis/index.html
> > GitHub: https://github.com/schuhschuh/cmake-basis/
> 
> The repo is surprisingly large (67 mb) for something like this. I consider
> such large files in a git repo to be bad practice. This is just the first
> thing I noticed...
> 
> Commit: [...]
> 
> Hmm, I hadn't noticed the size before. There are a few powerpoint presentations and a PDF manual that probably contribute to the size, there are also a couple of small external libraries that are directly committed into the tree. The zip version of the release is 2.7 MB which is not quite as bad. I'm not well versed with the guts of git but are there any good ways to do some cleanup?

My bad. The PowerPoint presentations were only on an internal Wiki server at the beginning, but having them within the repo which is supposed to document a standard (rather than being software alone) seemed to be the right place. Of course I would prefer now a different format than PowerPoint slides to create this kind of documentation. The PDF manual can be created from the reST source files and I was considering removing it again from version control for that very reason.

>  
> 
> > Also, if any CMake developers are interested in bringing any of this
> > functionality upstream into CMake itself we would also love to hear from
> > you.
> 
> I read some of the docs and some of the code (TargetTools.cmake). My
> understanding my not be complete or correct.
> 
> 1) BASIS seems to be doing lots of things. Can you identify what parts of
> BASIS or what concepts are most valuable/most useful/most missing from CMake
> etc? I think you could know better than we can.
> 
> I would suggest many of the items from the BASIS CMake modules, specifically the section labeled "BASIS Modules".
> 
> Some of the specific useful items include:
> basis_add_doc() - built in support of documentation tools
> Additional support for other languages, such as MATLAB and Python
> Automated packaging and installation of libraries
> Support for doxygen documentation of CMake Code
> I'm sure there is more but those are some of the most obvious items that I can think of off the top of my head.
>  
> For example, do you consider
> 
>  add_executable(myexe.cpp)
> 
> to be good user interface for creating a target called 'myexe', and
> something that CMake should do?
> 
> It seems like it would be pretty straightforward and easy to use. I've seen that functionality implemented independently on more than one occasion.

How often have you seen CMake code as the following

add_executable(foo foo.cpp)

?

The basis_add_executable supports this use case as well, but additionally the shorter notation (not counting the basis_ prefix ;) )

basis_add_executable(foo.cpp)

>  
> Should CMake automatically create
> installation rules for it?
> 
> I think this would be excellent if there was a reasonable default, plus more advanced configuration for when it is needed. This could also happen for libraries. Right now it seems CMake has a lot of boilerplate that could be skipped by making functions automate steps using reasonable defaults that can be modified or disabled as needed.
>  

Very often people add an executable target, build it, test it and use it all fine. But when they send out the packaged software, it happens that the executable is missing because the extra installation rule has not been added. Because the developer themselves mainly use the build tree directly, this issue simply never popped up before.

> 
> Another example: You have some target-source-file-globbing code. Should that
> be extracted, generalized and upstreamed?
> 
>  https://www.mail-archive.com/cmake-developers@cmake.org/msg06725.html
> 
> I've seen a number of requests for such funcitonality so it may be desirable, although I don't personally use it.

When I started at SBIA, they had a very simple CMake project template which was supposed to be as simple as any possible. For example, a developer would just put all Bash script files in one directory, and they would be auto-magically be configured/installed by CMake without them ever touching any CMakeLists.txt file. This was achieved using globbing expressions. The downside of this approach is well known. CMake does not recognize if files were added or removed and thus would not reconfigure itself. The BASIS extension of the basis_add_executable and basis_add_library commands is intended to bridge both, make the use of globbing expressions convenient and safe at the same time.

>  
> Another example: You have code for adding scripts as executables. What are
> the generic (non-BASIS related) use cases for that?
> 
> Many projects aren't libraries or installable software for distribution in one language, but instead a set of custom utilities that call each other via the general Unix philosophy. Also, groups that build embedded software or backend software will have scripts that automatically start, monitor, deploy, implement, and/or run their software. Programs based on scripting languages also need packaging and installation just like any other compiled language if they are to be opened via an icon on a user's system.

You can define dependencies among them or to modules written in the same scripting language. You can have find_package calls to find installed modules and declare that your script depends on them. An installation rule for the script may ensure that the modules get installed as well. When wrapping scripts in Windows Command files to mimic the she-bang directive of Unix, calling such script becomes just as convenient as it is on Unix from the command line without having to specify the interpreter executable explicitly.

>  
> Should CMake be extended with similar capabilities somehow?
> 
> I personally believe this could be very useful to quite a few users.
>  
> Would it help if an IMPORTED executable could be a script?
> 
> What would this entail? Is there perhaps another question behind this question?

The imported executable targets are used by BASIS to populate a mapping from target names to executable file paths. The idea is that one package may call executables of other packages, but refers to these only via their build target name which is supposed to not change over the lifetime of the (other) software. The executable file path is however dependent on installation, platform (e.g., .cmd on Windows, no extension on Unix),… further, it would be valid to prepend an executable output file name with a common package specific prefix to avoid conflicts with executables of other packages. Instead of relying on the actual name of the installed executable file, the build target is imported and any external executables are only called by their assigned build target name. The BASIS utility function “execute” which is provided for the different programming languages, takes care of mapping the build target name to the imported executable file path. See for example http://opensource.andreasschuh.com/cmake-basis/apidoc/latest/namespacebasis.html#a5ee690323f7927da88cc69179708cfaa (C++).

The advantageous are that the developer
- is independent of changes regarding the installed executable file name (either by author of other package or during build configuration)
- does not have to ensure a proper PATH, especially in case of multiple installed versions (which version will be used is specified at build configuration time)

Any paths to executables of the same package are relative to the location of the calling executable and it is thus guaranteed that the own auxiliary executables are being used, while paths to executables of other installed packages are absolute. Thus, the installation is independent of the PATH environment variable which can be tricky to set right especially with multiple versions of a software installed.

>  
> Another example: Can the documentation utilities be separated from the rest
> of BASIS, made generic and still be useful?
> 
> Yes, I believe that is possible. I may be mistaken but I believe DocTools.cmake is fairly independent aside from a few auxillary files in the local folder that it configures.
>  
> 
> 2) Do you generally recommend that CMake users use BASIS from now on, or is
> it something you expect will be niche/mostly used by yourselves?
> 
> Everyone will have different use cases and needs, and BASIS doesn't handle 100% of CMake functionality. However, I personally expect that I will use BASIS on my own projects because it simply cuts out a lot of the manual steps and boilerplate needed for documentation and setting up a typical project/library directory.

The first and foremost target group are clearly research groups and less experienced software developers or those who do not want to bother writing a considerable amount of CMake code rather than their actual software. The more BASIS matures and benefits from contributions of more experienced developers, the more useful it will be to a broader audience.

>  
> 
> 3) BASIS requires me to use basis_add_executable, KDE4 requires me to use
> kde4_add_executable, VTK requires me to use vtk_add_executable.
> 
> BASIS seems to wrap many or most CMake commands which define the
> buildsystem. That was realized to be bad practice in KDE. The new KDE
> Frameworks (KF5) do not have that requirement or practice, and instead the
> regular add_executable CMake commands (and similar) are recommended. CMake
> has been extended (by myself and others) to make that possible without
> losing features. 
> 
> This way, not only is interoperability increased, but users of a KF5 package
> can rely on the CMake documentation to understand and reason about code they
> read and write. Further, the buildsystem for KF5 is CMake not 'something on
> top of CMake', which then must be understood. Using CMake directly, and not
> 'something on top of CMake' is also a goal of the boost effort to migrate to
> CMake.

Reason for keeping all functions in their own universe is that they are not just doing about the same as the CMake commands, but extend and/or combine some of them to reduce the amount of CMake code needed in a project. Thus you cannot simply reuse the CMake documentation to describe their behavior. Initially I started off overriding the original CMake commands, but this transparency would in case of some of the BASIS CMake functions rather lead to more confusion and surprises, especially among experienced CMake users. Others, however, and this is the majority in my target group, is less familiar with CMake and further does not intend to learn more about it than is necessary. The only goal is to get the software compiled and linked. Teaching these fellows only few basic BASIS commands is easier than a number of CMake commands.

Another reason was to yet allow experienced developers the use of the standard CMake commands if they decided to do so (while loosing out on some of the features BASIS is offering, which however may not be needed/desired by these experienced fellows).

On a side note, just today a co-worker asked me why the compiler cannot find the header files when they were provided as additional arguments to the add_executable command. Indeed this was a reasonable assumption, I think. Why the need for an additional include_directories when I already specified where the header files are located ? This is not something BASIS is taking care of yet either, but would certainly be one of the things I would consider adding.

> 
> Further, wrappers like that can not cleanly offer all of the features of
> wrapped commands, as those commands get updated in CMake. I'm not sure your
> wrappers can handle ALIAS, INTERFACE or OBJECT libraries for example, and it
> seems that
> 
>  add_library(tgt SHARED IMPORTED)
> 
> might work, whereas
> 
>  add_library(tgt IMPORTED SHARED)
> 
> would not. Ok, it's not documented to work either, but my point is that
> wrappers are not good API proxies.
> 
> Any comments on that?
> 
> Cool, is there a good reference explaining how wrapped functions can be eliminated? Some of the functionality like the automated packaging steps are made possible by the wrapped functions, but if there was another way to do it I'm sure that would be ideal.
>  
> 
> 4) BASIS seems to expect settings, flags etc to be written as directory
> scoped. Modern CMake is increasingly scoped to targets, with directory
> scoped commands considered only convenience for populating properties on
> targets.
> 

This new functionality of CMake is very desirable indeed. However, convenience is all that matters to researchers. They seriously (for good reason) do not consider repeating themselves for each and every target but rather state these once at the top-level and keep it that way. If we can make the setting of these target properties more convenient without the need of too many extra CMake code for the users, that would be great.

>  http://www.cmake.org/cmake/help/git-master/manual/cmake-buildsystem.7.html#directory-scoped-commands
> 
> This is mostly helpful for transitive usage requirements and related CMake
> concepts.
> 
> Do you think BASIS would move in a similar direction?
> 
> Yes, I would expect it to follow the direction of CMake. As a side note, one thing I've wished for with CMake is an up-to-date style guide, how-to, and tutorials capturing the best recommended way to write Modern CMake code for a cross platform distributable library. For example, even thebasic cmake tutorial doesn't use the more recent "Modern CMake" capabilities.
>  
> 
> 
> Thanks,
> 
> Steve.
>  
> Thanks for all your feedback and interest! It is appreciated.
> -- 
> 
> Powered by www.kitware.com
> 
> Please keep messages on-topic and check the CMake FAQ at: http://www.cmake.org/Wiki/CMake_FAQ
> 
> Kitware offers various services to support the CMake community. For more information on each offering, please visit:
> 
> CMake Support: http://cmake.org/cmake/help/support.html
> CMake Consulting: http://cmake.org/cmake/help/consulting.html
> CMake Training Courses: http://cmake.org/cmake/help/training.html
> 
> Visit other Kitware open-source projects at http://www.kitware.com/opensource/opensource.html
> 
> Follow this link to subscribe/unsubscribe:
> http://www.cmake.org/mailman/listinfo/cmake

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.cmake.org/pipermail/cmake/attachments/20140123/4bfe53e2/attachment-0001.html>