[vtk-developers] [ANNOUNCE] New module system landing

Ben Boeckel ben.boeckel at kitware.com
Wed Jan 2 14:23:53 EST 2019

Hi all,

The new module system is in a landable state right now. We're planning
on merging it Friday (4 Jan 2019). The buildbots will be updated shortly
afterwards. Once they've been updated, `Do: test` will only work for
rebased merge requests (since the cache entries don't line up). In
addition, after the update, buildbots will default to building all
modules (turning off those which don't build).

The CMake API is now much more regular (some internal APIs have an
oddity or two we can clean up). Keyword arguments, erroring out on
unrecognized arguments, proper error messages for missing required
arguments, etc. All API functions also have a documentation block above
them in Markdown. In the future, this will be extracted and added to the
Doxygen output for VTK. In addition, no global variables[1] are used to
control the build; everything is passed in via arguments to the relevant
functions. There is one slight exception to this for APIs meant to be
called within another context. For example, `vtk_module_add_module` uses
the `LIBRARY_DESTINATION` variable passed to `vtk_module_build` for its
installation. The testing CMake APIs have a few similar expectations.

Previous discussion has occured on this thread on this list:


The merge request is here:


ParaView's update for the new build setup will be landing in the
following days.

Changes for building VTK itself:

  - CMake 3.8 is now required. This is primarily to support
    `target_compile_features(cxx_std_11)` for having *consumers* of VTK
    also have (at least) C++11 enabled. Building with kits requires
    3.12 for object library features. Some modules require newer CMake
    versions for builtin CUDA language support (though they don't seem
    to be guarded by checks at the moment).
  - Module names have been changed. For example, `vtkCommonCore` is now
    `VTK::CommonCore`. This allows for one name to work both inside and
    outside VTK for `target_link_libraries`. Cache variable names use
    the module name.
  - Changed cache variables:
    * Module and group flags:
      - `Module_<MMM>` -> `VTK_MODULE_ENABLE_<MMM>`
      - `VTK_Group_<MMM>` -> `VTK_GROUP_ENABLE_<MMM>`
      - Values are no longer boolean, but instead 5-state:
        * `YES`: Must be built
        * `NO`: Must not be built (turns off modules requiring it;
          errors if required by a `YES` module)
        * `WANT`: Build if possible
        * `DONT_WANT`: Build only if necessary
        * `DEFAULT`: Use group (for modules) and `VTK_BUILD_ALL_MODULES`
        Third party modules which don't support an external version no
        longer have an option available.
  - Turning off Python forces Python modules off.
  - Turning off Java forces Java modules off.
  - Turning off MPI forces MPI modules off.
  - Configuring VTK is now faster. An "all modules" build with Python
    wrapping enabled:

      cmake .  4.68s user 0.84s system 98% cpu 5.578 total

    Doing this on `master` on the same machine and Python *off*:

      cmake .  25.44s user 19.79s system 99% cpu 45.297 total

    This is not a scientific comparison; the module set is as close as I
    could make it, but if anything, `master` has fewer modules enabled
    (find modules not working, Python being disabled, some modules are
    new due to splitting, etc.).
  - CMake output is now minimal. Here is the complete output from
    reconfiguring VTK:

    % cmake .
    -- HDF5: Using hdf5 compiler wrapper to determine C configuration
    -- Configuring done
    -- Generating done
    -- Build files have been written to: /home/boeckb/code/depot/group-kitware/vtk/build

    Not sure where the HDF5 output is coming from; might be a bug in
  - The VTKm submodule has moved one directory lower (it is now
    ThirdParty/vtkm/vtkvtkm/vtk-m so that the settings for a local VTKm
    are localized).

For development:

  - Adding a module is similar to before. See `vtk.module` files in the
    source tree.
  - All files should be included in the source listing for modules.
    Using `CLASSES` is the same as doing `<NAME>.cxx` as a source and
    `<NAME>.h` as a public header. Private headers should use
    `PRIVATE_HEADERS` (e.g., object factory files).
  - To hide a module from a build (e.g., a Windows-only module), the
    `CONDITION` argument should be used in the `vtk.module` file. This
    is done for the Python and Java modules.
  - The object factory API has also been updated to the same patterns as
    the new module system (output variables rather than directory
    properties). See existing code for usage.
  - The new CMake API only works when controlled via the
    `vtk_module_scan` and `vtk_module_build` functions. Global and
    target properties are used to store information, so
    `vtk_module_add_module` doesn't work with just a simple
  - Due to kits being a thing, the backing target for a module may have
    different names based of CMake flags. Basically, CMake's `target_`
    commands need to be given `CommonCore`, `VTK::CommonCore`, or
    `CommonCore-objects` based on the build settings. To simplify this,
    there are functions like `vtk_module_include` or `vtk_module_link`
    for handling modules. Similarly for properties, there is
  - Prefer imported targets rather than `INCLUDE_DIRS` and `LIBRARIES`
    variables. This ensures that usage requirements are properly done
    and that VTK is relocatable once installed.
  - Public cache variables should be placed as close to their usage as
    possible. For example, the OpenGL settings are now in
    `Utilities/OpenGL` rather than the top level.

For consumers of VTK:

  - `find_package(VTK COMPONENTS CommonCore)` should be used.
    `OPTIONAL_COMPONENTS` is also supported.
  - Install tree is largely the same layout (library names are still
  - For object factories to work, `vtk_module_autoinit` should be used.
    This generates the proper `-D` flags for getting initialization to
  - Language wrapping APIs is now *much* simpler. Here is the call to
    wrap VTK's modules in Python:

        MODULES         ${vtk_modules}
        PYTHON_PACKAGE  "vtkmodules"
        CMAKE_DESTINATION   "${vtk_cmake_destination}"
        WRAPPED_MODULES vtk_python_wrapped_modules
        TARGET          VTK::vtkpythonmodules)

    and Java:

        MODULES         ${vtk_modules}
        WRAPPED_MODULES vtk_java_wrapped_modules)

  - No more PythonD libraries. VTK modules now import their dependent
    modules automatically rather than linking to the backing
    implementations directly (mainly a packager detail, but important).

Things I've noticed on this adventure:

  - To default a module to being built, add it to a group that is on by
    default (currently just `StandAlone` and `Rendering`). Excess
    dependencies is not the way to do this.
  - There are a few modules with debug code which uses modules not in
    the dependency list. Debug code should constrain itself to the same
    dependencies the rest of the code uses (or have just a comment in
    the `vtk.module` file as to its use).

Current state:

  - Fewer than 25 (out of 2000+) tests failing on a build everything[2]
    build with Python enabled.
  - Python optional linking is currently disabled. On Linux, linking
    errors cropped up and macOS had runtime errors appear. Investigation
    is needed.
  - Some examples disabled (Medical and Modelling are ported, but
    *their* tests fail right now). Those with `UNMAINTAINED.md` files
    are *not* ported.
  - The Medical and Modelling examples are disabled. They run tests
    during which fail. Investigation is nedded.
  - Debug points need to be added. However, I'd like to see where
    they're commonly necessary before adding them everywhere they
    *might* be useful. Places I can forsee it being nice to have:
    * Why is this test module not enabled?
    * Why is this module built?
    * Why is this module not built?
  - Remote modules have not been ported nor are the download/build
    mechanisms wired up.
  - There seems to be some issue with finding GLX on some Ubuntu
    installs with custom nvidia driver installations. It's likely
    something wrong with FindOpenGL, but more information needs to be

Near future tasks:

  - The new module system now supports optional test dependencies.
    Moving some test dependencies to be optional and only building the
    parts of the test suite that can would help improve how much of the
    test suite runs in smaller builds.
  - Excess data. There is data that nothing uses (input and baselines).
    This should be sorted through and we should figure out whether they
    were missed when removing old tests or new tests should be written
    to use the data.
  - Updating WikiExamples to use the new system.

Further future tasks:

  - Update Python code to `from vtkmodules.MMM import XXX` instead of
    `from vtk import XXX`. This should vastly improve Python test times.
  - Reducing the number of `try_compile` performed. This will immensely
    help with initial configure times. One idea is to port third party
    libraries over to using KWIML for builtin types rather than checking
    their sizes and signedness at configure time. Upstreamable solutions
    are preferred however.

Third party packages in VTK have also been updated. In general, they no
longer add (visible) cache entries at all and their `try_compile` tests
have been minimized. Here's the current listing of internal third party
libraries in VTK:

  diy2-3.5.0 (I)            libproj 4.9.3       tiff 4.0.6
  doubleconversion 3.1.1    libxml2 2.9.8       utf8 2.3.4 (I)
  eigen 3.3.5               lz4 1.8.3           verdict 1.2.0 (IC)
  exodusII 7.15f (I)        lzma 5.2.4          vpic ???
  expat 2.2.6               mpi4py 2.0.0        vtk-m <master> (I)
  freetype 2.9.1            netcdf 4.6.1        xdmf2 1.2.11 (I)
  gl2ps 1.4.0 (P)           ogg 1.3.3           xdmf3 1.2.11 (I)
  glew 2.1.0                pegtl 2.7.1 (I)     zfp 0.5.4 (I)
  hdf5 1.10.3               png 1.6.35          zlib 1.2.11
  jpeg 2.0.0                pugixml 1.9         kwiml <master> (I)
  jsoncpp 1.8.4             sqlite 3.25.2       kwsys <master> (I)
  kissfft ??? (I)           theora 1.1.1        metaio <master> (I)
  libharu 2.4.0 (P)

  - `C` means it needs converted to use `update.sh` yet
  - `I` means external versions are not supported
  - `P` means upstream needs to merge some PRs before an external
    version would actually work (links to PRs in the relevant CMake

CDash will now be showing all warnings again (I cleaned out the regular
expressions since third party libraries have been uplifted). I've also
pushed many GCC 8 warning fixes upstream, so MSVC should be the only one
with oodles of warnings yet. When adding exclusions back, please include
comments as to why they are there and, if applicable, a link to an issue
about actually addressing them.


[1]There are a few global variables that are read, but they only control
*defaults* to cache variables, not the variable itself (e.g., defaulting
StandAlone and Rendering groups to be built).

[2]Some modules were excluded from this testing on some platforms due to
a lack of dependencies on the testing machines:

    VTK::FiltersOpenTurns     VTK::RenderingOculus
    VTK::IOADIOS              VTK::RenderingOpenVR
    VTK::IOPDAL               VTK::RenderingOptiX
    VTK::FiltersOpenTurns     VTK::RenderingOSPRay
    VTK::IOADIOS              VTK::RenderingOculus
    VTK::IOMySQL              VTK::RenderingOpenVR
    VTK::IOPDAL               VTK::RenderingOptiX
    VTK::IOPostgreSQL         VTK::mpi
    VTK::DomainsMicroscopy    VTK::IOPostgreSQL
    VTK::FiltersOpenTurns     VTK::InfovisBoost
    VTK::FiltersReebGraph     VTK::InfovisBoostGraphAlgorithms
    VTK::GUISupportMFC        VTK::Java
    VTK::IOADIOS              VTK::RenderingExternal
    VTK::IOFFMPEG             VTK::RenderingFreeTypeFontConfig
    VTK::IOGDAL               VTK::RenderingOSPRay
    VTK::IOIODBC              VTK::RenderingOculus
    VTK::IOLAS                VTK::RenderingOpenVR
    VTK::IOPDAL               VTK::RenderingOptiX
    VTK::IOPDAL               VTK::xdmf3

Code itself was not touched except where errors were otherwise found.

More information about the vtk-developers mailing list