[vtk-developers] [ANNOUNCE] New module system landing
Ben Boeckel
ben.boeckel at kitware.com
Wed Jan 2 14:23:53 EST 2019
Hi all,
The new module system is in a landable state right now. We're planning
on merging it Friday (4 Jan 2019). The buildbots will be updated shortly
afterwards. Once they've been updated, `Do: test` will only work for
rebased merge requests (since the cache entries don't line up). In
addition, after the update, buildbots will default to building all
modules (turning off those which don't build).
The CMake API is now much more regular (some internal APIs have an
oddity or two we can clean up). Keyword arguments, erroring out on
unrecognized arguments, proper error messages for missing required
arguments, etc. All API functions also have a documentation block above
them in Markdown. In the future, this will be extracted and added to the
Doxygen output for VTK. In addition, no global variables[1] are used to
control the build; everything is passed in via arguments to the relevant
functions. There is one slight exception to this for APIs meant to be
called within another context. For example, `vtk_module_add_module` uses
the `LIBRARY_DESTINATION` variable passed to `vtk_module_build` for its
installation. The testing CMake APIs have a few similar expectations.
Previous discussion has occured on this thread on this list:
https://markmail.org/thread/cu2wyqpvporqvps6
The merge request is here:
https://gitlab.kitware.com/vtk/vtk/merge_requests/5020
ParaView's update for the new build setup will be landing in the
following days.
Changes for building VTK itself:
- CMake 3.8 is now required. This is primarily to support
`target_compile_features(cxx_std_11)` for having *consumers* of VTK
also have (at least) C++11 enabled. Building with kits requires
3.12 for object library features. Some modules require newer CMake
versions for builtin CUDA language support (though they don't seem
to be guarded by checks at the moment).
- Module names have been changed. For example, `vtkCommonCore` is now
`VTK::CommonCore`. This allows for one name to work both inside and
outside VTK for `target_link_libraries`. Cache variable names use
the module name.
- Changed cache variables:
* Module and group flags:
- `Module_<MMM>` -> `VTK_MODULE_ENABLE_<MMM>`
- `VTK_Group_<MMM>` -> `VTK_GROUP_ENABLE_<MMM>`
- Values are no longer boolean, but instead 5-state:
* `YES`: Must be built
* `NO`: Must not be built (turns off modules requiring it;
errors if required by a `YES` module)
* `WANT`: Build if possible
* `DONT_WANT`: Build only if necessary
* `DEFAULT`: Use group (for modules) and `VTK_BUILD_ALL_MODULES`
instead
- `VTK_USE_SYSTEM_<MMM>` -> `VTK_MODULE_USE_EXTERNAL_<MMM>`
Third party modules which don't support an external version no
longer have an option available.
- Turning off Python forces Python modules off.
- Turning off Java forces Java modules off.
- Turning off MPI forces MPI modules off.
- Configuring VTK is now faster. An "all modules" build with Python
wrapping enabled:
cmake . 4.68s user 0.84s system 98% cpu 5.578 total
Doing this on `master` on the same machine and Python *off*:
cmake . 25.44s user 19.79s system 99% cpu 45.297 total
This is not a scientific comparison; the module set is as close as I
could make it, but if anything, `master` has fewer modules enabled
(find modules not working, Python being disabled, some modules are
new due to splitting, etc.).
- CMake output is now minimal. Here is the complete output from
reconfiguring VTK:
% cmake .
-- HDF5: Using hdf5 compiler wrapper to determine C configuration
-- Configuring done
-- Generating done
-- Build files have been written to: /home/boeckb/code/depot/group-kitware/vtk/build
Not sure where the HDF5 output is coming from; might be a bug in
`FindHDF5`.
- The VTKm submodule has moved one directory lower (it is now
ThirdParty/vtkm/vtkvtkm/vtk-m so that the settings for a local VTKm
are localized).
For development:
- Adding a module is similar to before. See `vtk.module` files in the
source tree.
- All files should be included in the source listing for modules.
Using `CLASSES` is the same as doing `<NAME>.cxx` as a source and
`<NAME>.h` as a public header. Private headers should use
`PRIVATE_HEADERS` (e.g., object factory files).
- To hide a module from a build (e.g., a Windows-only module), the
`CONDITION` argument should be used in the `vtk.module` file. This
is done for the Python and Java modules.
- The object factory API has also been updated to the same patterns as
the new module system (output variables rather than directory
properties). See existing code for usage.
- The new CMake API only works when controlled via the
`vtk_module_scan` and `vtk_module_build` functions. Global and
target properties are used to store information, so
`vtk_module_add_module` doesn't work with just a simple
`add_subdirectory`).
- Due to kits being a thing, the backing target for a module may have
different names based of CMake flags. Basically, CMake's `target_`
commands need to be given `CommonCore`, `VTK::CommonCore`, or
`CommonCore-objects` based on the build settings. To simplify this,
there are functions like `vtk_module_include` or `vtk_module_link`
for handling modules. Similarly for properties, there is
`vtk_module_set_property`.
- Prefer imported targets rather than `INCLUDE_DIRS` and `LIBRARIES`
variables. This ensures that usage requirements are properly done
and that VTK is relocatable once installed.
- Public cache variables should be placed as close to their usage as
possible. For example, the OpenGL settings are now in
`Utilities/OpenGL` rather than the top level.
For consumers of VTK:
- `find_package(VTK COMPONENTS CommonCore)` should be used.
`OPTIONAL_COMPONENTS` is also supported.
- Install tree is largely the same layout (library names are still
`vtkCommonCore-X.Y`).
- For object factories to work, `vtk_module_autoinit` should be used.
This generates the proper `-D` flags for getting initialization to
work.
- Language wrapping APIs is now *much* simpler. Here is the call to
wrap VTK's modules in Python:
vtk_module_wrap_python(
MODULES ${vtk_modules}
INSTALL_EXPORT VTK
PYTHON_PACKAGE "vtkmodules"
MODULE_DESTINATION "${VTK_PYTHON_SITE_PACKAGES_SUFFIX}"
CMAKE_DESTINATION "${vtk_cmake_destination}"
WRAPPED_MODULES vtk_python_wrapped_modules
TARGET VTK::vtkpythonmodules)
and Java:
vtk_module_wrap_java(
JAVA_OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/src/vtk"
MODULES ${vtk_modules}
WRAPPED_MODULES vtk_java_wrapped_modules)
- No more PythonD libraries. VTK modules now import their dependent
modules automatically rather than linking to the backing
implementations directly (mainly a packager detail, but important).
Things I've noticed on this adventure:
- To default a module to being built, add it to a group that is on by
default (currently just `StandAlone` and `Rendering`). Excess
dependencies is not the way to do this.
- There are a few modules with debug code which uses modules not in
the dependency list. Debug code should constrain itself to the same
dependencies the rest of the code uses (or have just a comment in
the `vtk.module` file as to its use).
Current state:
- Fewer than 25 (out of 2000+) tests failing on a build everything[2]
build with Python enabled.
- Python optional linking is currently disabled. On Linux, linking
errors cropped up and macOS had runtime errors appear. Investigation
is needed.
- Some examples disabled (Medical and Modelling are ported, but
*their* tests fail right now). Those with `UNMAINTAINED.md` files
are *not* ported.
- The Medical and Modelling examples are disabled. They run tests
during which fail. Investigation is nedded.
- Debug points need to be added. However, I'd like to see where
they're commonly necessary before adding them everywhere they
*might* be useful. Places I can forsee it being nice to have:
* Why is this test module not enabled?
* Why is this module built?
* Why is this module not built?
- Remote modules have not been ported nor are the download/build
mechanisms wired up.
- There seems to be some issue with finding GLX on some Ubuntu
installs with custom nvidia driver installations. It's likely
something wrong with FindOpenGL, but more information needs to be
gathered.
Near future tasks:
- The new module system now supports optional test dependencies.
Moving some test dependencies to be optional and only building the
parts of the test suite that can would help improve how much of the
test suite runs in smaller builds.
- Excess data. There is data that nothing uses (input and baselines).
This should be sorted through and we should figure out whether they
were missed when removing old tests or new tests should be written
to use the data.
- Updating WikiExamples to use the new system.
Further future tasks:
- Update Python code to `from vtkmodules.MMM import XXX` instead of
`from vtk import XXX`. This should vastly improve Python test times.
- Reducing the number of `try_compile` performed. This will immensely
help with initial configure times. One idea is to port third party
libraries over to using KWIML for builtin types rather than checking
their sizes and signedness at configure time. Upstreamable solutions
are preferred however.
Third party packages in VTK have also been updated. In general, they no
longer add (visible) cache entries at all and their `try_compile` tests
have been minimized. Here's the current listing of internal third party
libraries in VTK:
diy2-3.5.0 (I) libproj 4.9.3 tiff 4.0.6
doubleconversion 3.1.1 libxml2 2.9.8 utf8 2.3.4 (I)
eigen 3.3.5 lz4 1.8.3 verdict 1.2.0 (IC)
exodusII 7.15f (I) lzma 5.2.4 vpic ???
expat 2.2.6 mpi4py 2.0.0 vtk-m <master> (I)
freetype 2.9.1 netcdf 4.6.1 xdmf2 1.2.11 (I)
gl2ps 1.4.0 (P) ogg 1.3.3 xdmf3 1.2.11 (I)
glew 2.1.0 pegtl 2.7.1 (I) zfp 0.5.4 (I)
hdf5 1.10.3 png 1.6.35 zlib 1.2.11
jpeg 2.0.0 pugixml 1.9 kwiml <master> (I)
jsoncpp 1.8.4 sqlite 3.25.2 kwsys <master> (I)
kissfft ??? (I) theora 1.1.1 metaio <master> (I)
libharu 2.4.0 (P)
- `C` means it needs converted to use `update.sh` yet
- `I` means external versions are not supported
- `P` means upstream needs to merge some PRs before an external
version would actually work (links to PRs in the relevant CMake
file)
CDash will now be showing all warnings again (I cleaned out the regular
expressions since third party libraries have been uplifted). I've also
pushed many GCC 8 warning fixes upstream, so MSVC should be the only one
with oodles of warnings yet. When adding exclusions back, please include
comments as to why they are there and, if applicable, a link to an issue
about actually addressing them.
--Ben
[1]There are a few global variables that are read, but they only control
*defaults* to cache variables, not the variable itself (e.g., defaulting
StandAlone and Rendering groups to be built).
[2]Some modules were excluded from this testing on some platforms due to
a lack of dependencies on the testing machines:
Linux:
VTK::FiltersOpenTurns VTK::RenderingOculus
VTK::IOADIOS VTK::RenderingOpenVR
VTK::IOPDAL VTK::RenderingOptiX
VTK::RenderingOSPRay
macOS:
VTK::FiltersOpenTurns VTK::RenderingOSPRay
VTK::IOADIOS VTK::RenderingOculus
VTK::IOMySQL VTK::RenderingOpenVR
VTK::IOPDAL VTK::RenderingOptiX
VTK::IOPostgreSQL VTK::mpi
VTK::Java
Windows:
VTK::DomainsMicroscopy VTK::IOPostgreSQL
VTK::FiltersOpenTurns VTK::InfovisBoost
VTK::FiltersReebGraph VTK::InfovisBoostGraphAlgorithms
VTK::GUISupportMFC VTK::Java
VTK::IOADIOS VTK::RenderingExternal
VTK::IOFFMPEG VTK::RenderingFreeTypeFontConfig
VTK::IOGDAL VTK::RenderingOSPRay
VTK::IOIODBC VTK::RenderingOculus
VTK::IOLAS VTK::RenderingOpenVR
VTK::IOPDAL VTK::RenderingOptiX
VTK::IOPDAL VTK::xdmf3
Code itself was not touched except where errors were otherwise found.
More information about the vtk-developers
mailing list