[vtk-developers] [ANNOUNCE] New module system landing

[Andras Lasso]

I just wanted to clarify that application developers only need YES and DONT_WANT states for VTK modules. If the other states are required internally in VTK (for example, for VTK automatic tests) then it's fine, but please try to keep the VTK's public CMake configuration API simple.

> Right now it says "X requires disabled module Y". What's missing is information of the link between X and Y for figuring out how to fix it.

Dependency graph printing feature of CTK fixes this. See an example below how external project dependency graph is printed during application configuration. @jcfr can give you more details on how this works and could be adapted to VTK modules.

Andras


External project dependency graph printed during CMake configuration of Slicer:

-- SuperBuild - Slicer => Requires curl, CTKAppLauncherLib, teem, VTK, ITK, CTK, LibArchive, RapidJSON, SimpleITK, SlicerExecutionModel, qRestAPI, DCMTK, python-pydicom, python-packaging, python-chardet, python-couchdb, python-GitPython, python-pip, python-six, python-PyGithub, CTKAPPLAUNCHER, python, NUMPY, tbb[INCLUDED], 
-- SuperBuild -   curl => Requires zlib, OpenSSL, 
-- SuperBuild -     zlib[OK]
-- SuperBuild -     OpenSSL[OK]
-- SuperBuild -   curl[OK]
-- SuperBuild -   CTKAppLauncherLib[OK]
-- SuperBuild -   teem => Requires zlib[INCLUDED], VTK, 
-- SuperBuild -     VTK => Requires zlib[INCLUDED], python, tbb[INCLUDED], 
-- SuperBuild -       python => Requires bzip2, CTKAPPLAUNCHER, zlib[INCLUDED], OpenSSL[INCLUDED], 
-- SuperBuild -         bzip2[OK]
-- SuperBuild -         CTKAPPLAUNCHER => Requires CTKResEdit, 
-- SuperBuild -           CTKResEdit[OK]
-- SuperBuild -         CTKAPPLAUNCHER[OK]
-- SuperBuild -       python[OK]
-- SuperBuild -     VTK[OK]
-- SuperBuild -   teem[OK]
-- SuperBuild -   ITK => Requires zlib[INCLUDED], VTK[INCLUDED], DCMTK, 
-- SuperBuild -     DCMTK => Requires zlib[INCLUDED], 
-- SuperBuild -     DCMTK[OK]
-- SuperBuild -   ITK[OK]
-- SuperBuild -   CTK => Requires VTK[INCLUDED], ITK[INCLUDED], python[INCLUDED], DCMTK[INCLUDED], 
-- SuperBuild -   CTK[OK]
-- SuperBuild -   LibArchive => Requires zlib[INCLUDED], zlib[INCLUDED], 
-- SuperBuild -   LibArchive[OK]
-- SuperBuild -   RapidJSON[OK]
-- SuperBuild -   SimpleITK => Requires ITK[INCLUDED], Swig[INCLUDED], python[INCLUDED], python-setuptools, 
-- SuperBuild -     python-setuptools => Requires python[INCLUDED], 
-- SuperBuild -     python-setuptools[OK]
-- SuperBuild -   SimpleITK[OK]
-- SuperBuild -   SlicerExecutionModel => Requires ITK[INCLUDED], JsonCpp, ParameterSerializer, 
-- SuperBuild -     JsonCpp[OK]
-- SuperBuild -     ParameterSerializer => Requires JsonCpp[INCLUDED], ITK[INCLUDED], 
-- SuperBuild -     ParameterSerializer[OK]
-- SuperBuild -   SlicerExecutionModel[OK]
-- SuperBuild -   qRestAPI[OK]
-- SuperBuild -   python-pydicom => Requires python[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -   python-pydicom[OK]
-- SuperBuild -   python-packaging => Requires python[INCLUDED], python-pyparsing, 
-- SuperBuild -     python-pyparsing => Requires python[INCLUDED], 
-- SuperBuild -     python-pyparsing[OK]
-- SuperBuild -   python-packaging[OK]
-- SuperBuild -   python-chardet => Requires python[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -   python-chardet[OK]
-- SuperBuild -   python-couchdb => Requires python[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -   python-couchdb[OK]
-- SuperBuild -   python-GitPython => Requires python[INCLUDED], python-setuptools[INCLUDED], python-gitdb, 
-- SuperBuild -     python-gitdb => Requires python[INCLUDED], python-setuptools[INCLUDED], python-smmap, 
-- SuperBuild -       python-smmap => Requires python[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -       python-smmap[OK]
-- SuperBuild -     python-gitdb[OK]
-- SuperBuild -   python-GitPython[OK]
-- SuperBuild -   python-pip => Requires python[INCLUDED], python-wheel, 
-- SuperBuild -     python-wheel => Requires python[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -     python-wheel[OK]
-- SuperBuild -   python-pip[OK]
-- SuperBuild -   python-six => Requires python[INCLUDED], 
-- SuperBuild -   python-six[OK]
-- SuperBuild -   python-PyGithub => Requires python[INCLUDED], python-setuptools[INCLUDED], python-pyjwt, 
-- SuperBuild -     python-pyjwt => Requires python[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -     python-pyjwt[OK]
-- SuperBuild -   python-PyGithub[OK]
-- SuperBuild -   NUMPY => Requires python[INCLUDED], python-setuptools[INCLUDED], python-nose, 
-- SuperBuild -     python-nose => Requires python[INCLUDED], python-setuptools[INCLUDED], 
-- SuperBuild -     python-nose[OK]
-- SuperBuild -   NUMPY[OK]
-- SuperBuild - Slicer[OK]
-- Configuring done
-- Generating done


-----Original Message-----
From: Ben Boeckel <ben.boeckel at kitware.com> 
Sent: Monday, February 4, 2019 9:49 AM
To: Andras Lasso <lasso at queensu.ca>
Cc: David Thompson <david.thompson at kitware.com>; VTK Developers <vtk-developers at vtk.org>; David Gobbi <david.gobbi at gmail.com>
Subject: Re: [vtk-developers] [ANNOUNCE] New module system landing

On Sun, Feb 03, 2019 at 12:09:10 +0000, Andras Lasso wrote:
> I've used two large CMake-based projects in the past that tried to do 
> something similar - guess which features to enable/disable based on a 
> number of require/prohibit CMake flags. Both worked very poorly, 
> because it always required lengthy experimentation with the flags to 
> get the list of features that we  actually needed.

With the VTK module system, you say `YES` to modules you require. That guarantees you the features you want. You may optionally say `NO` to modules that you don't want for some reason (e.g., MPI or Python) which guarantees that they do not build even via dependency resolution.

> The main issue is that positive (require) and negative (prohibit) 
> requirements are obviously often contradict and it is not clear which 
> one is stronger, why exactly they conflict, and what are the options 
> to resolve the conflict.  Weak require and weak prohibit further 
> complicate things with questionable benefits.

Both are exactly as strong and the module system refuses to continue if the graph containing either cannot be made disjoint (mainly by disabling testing bits of modules). Having only strong YES/NO makes it easy to accidentally create a situation where I can't build anything without turning MPI on (currently a supported configuration) since I can't say "turn off MPI completely" if anything requires it since there's no "weak requires" semantics.

> I would recommend to reconsider the introduction of this unusual and 
> complicated 5-state system and only have binary REQUIRE on/off flag 
> for each module. The build system would only include required modules 
> and their prerequisites (everything else would be disabled).

This is what we had before and telling VTK to *not* build something (e.g., I don't have an external dependency) was painful since it was only ever a weak disable.

> If there
> was any conflict of required modules then the build system would print 
> an error message explaining why the combination is not feasible and 
> the the developer would decide how to modify the requirements.

Right now it says "X requires disabled module Y". What's missing is information of the link between X and Y for figuring out how to fix it.

> Instead of VTK checking PROHIBITED modules, the external project that 
> uses VTK can inspect the list of modules and throw a fatal CMake error 
> if it finds that a prohibited module got enabled (it can display a 
> much more informative message than VTK could, explaining why that 
> module is not supported and what to do about it). ATTEMPT and AVOID 
> can be similarly handled in the external project by logging a warning 
> if the resulting module list is not optimal.

Why would any project care that VTK has modules that it doesn't use? The only reason I can think of off-hand is a project not caring about such things would be if it is providing conflicting symbols (e.g., the same
class) that VTK does in one of those modules. That's already fragile and broken. For object factory issues, this issue should resolve things like
that:

    https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.kitware.com%2Fvtk%2Fvtk%2Fissues%2F17218&data=02%7C01%7Classo%40queensu.ca%7C40363f7095fe40e5ee1f08d68aaff357%7Cd61ecb3b38b142d582c4efb2838b925c%7C1%7C0%7C636848885639538130&sdata=fs6slxAXqlZDD79GDuxiqBKeEbz0Vic2EPQyEY6Mzj4%3D&reserved=0

> During configuration, a dependency graph can be printed (already 
> available in CTK and used in most CTK-based projects for external 
> project dependencies) to show how a component brings in other 
> components. This helps the developer in deciding which modules to 
> disable to get rid of prohibited or undesired modules.

Better debugging is on my list, but other tasks have higher priority right now (though I can review other MRs doing this).

--Ben