[CMake] Building third party libraries along with normal targets

[Elizabeth A. Fischer]

Miller,

Thank you for your input; I'm hoping we can use it to improve our
description of Spack at http://spack.io  .  I'm cross-posting to the Spack
list, maybe someone there can add to this.

https://github.com/LLNL/spack/issues/2115

There has been talk of a comparison; but I'm not familiar enough with the
other systems to talk about them.  I can/will say a bit about Spack.


>
but none actually have defined the problem or their solution.


Spack is aimed at people involved in scientific computing.  This is more a
social distinction than technical: there is nothing particularly special
about scientific software environments, and I know of corporate
environments that are considering Spack as well.  However, this intended
audience does drive the design in some ways.

1. As with Macports, Homebrew, Gentoo, etc. Spack is focused on building
things from source.  There has been some work on packing up Spack builds as
RPMs so they can be quickly installed across multiple machines.

2. Spack supports cross-compilers.  Cross compilation is an intrinsic part
of some supercomputers, in which the login nodes are a different
architecture from the nodes the software runs on.  Maybe a Spacker who has
done cross-compiling can speak up here...

3. Spack supports multiple versions of installed packages --- where the
"version" is actually a hash of the package version, plus the version of
all its dependencies.  Central to Spack is the "concretization" algorithm,
which chooses which version of each package to build for you, based on the
constraints you give it.  For example, you could build 24 versions of
NetCDF, based on different compilers and MPI options you choose.  This kind
of flexibility is incredibly useful in a scientific environment, where
different top-level applications require different versions of various
underlying dependencies.
        AFAIK, this concretization algorithm unique among auto-builders;
and it deserves some time to think about (look at the Powerpoint
presentations on the Spack page).  More common are builders (eg, those used
for Linux distros) that produce a *single* version of each desired package
--- and you have to manually specify the version of each package you want.
This is the step that Spack does automatically for you.



> what packages they create,


Spack now has over 1000 packages it knows how to build:

https://github.com/LLNL/spack/tree/develop/var/spack/repos/builtin/packages

NOTE: Be careful when comparing how many packages an auto-builder can
create.  Some package managers, for example, have a different "package" for
each version that might be installed.  For example, *netcdf-1.0*,
*netcdf-1.1*, *netcdf-1.2*, etc.  In Spack, each "package" corresponds to
an upstream piece of software (i.e. just *netcdf)*, and Spack knows how to
install potentially many versions and variants of it*.*


> what compromises they make,



> what they expect of my environment...


Just Python 2.6 or 2.7: do a "git clone", then run, starting up is very
easy.  Spack needs other basic tools as well (*git*, *curl*, etc.).  On
newer systems, they "just work"; on older systems, Spack can be used to
install them.


limitations


1. Spack currently runs on Linux + MacOS, not Windows.

2. Spack doesn't really know how to work with sub-package managers: eg,
using Spack to install `pip`, and then using `pip` to install things.  Same
thing for `luarocks`, `conda`, `node-js`, etc.  Spack *can* install these
things, just don't expect great integration.  We've thought a lot about how
to integrate well, and haven't come up with any home run ideas.  The issue
is most significant with *node-js* when a Spack package wants to depend on
a bunch of Javascript packages.

3. Spack's concretization algorithm, while still much faster than a human,
is too slow to conveniently use for software stacks of greater than ~100
packages.  This also makes it less than ideal for the Javascript world,
where packages are small and numerous.


> the other 10% of the time our system has an imcompatible version of some
> library and things blow up when you try to use some uncommon feature).


Spack goes to great lengths to build all dependencies itself, and not
"accidentally" depend on system-supplied software.

This is getting off-topic for the cmake mailing list, but I don't know
> where else to move the conversation.
>

Feel free to join the Spack mailing list (CC'd above).

-- Elizabeth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://public.kitware.com/pipermail/cmake/attachments/20170330/c9365492/attachment.html>