[vtk-developers] [ANNOUNCE] New module system landing

[Ben Boeckel]

On Tue, Jan 08, 2019 at 11:56:48 -0500, Ben Boeckel wrote:
> On Wed, Jan 02, 2019 at 14:23:53 -0500, Ben Boeckel wrote:
> >       - `Module_<MMM>` -> `VTK_MODULE_ENABLE_<MMM>`
> >       - `VTK_USE_SYSTEM_<MMM>` -> `VTK_MODULE_USE_EXTERNAL_<MMM>`
> 
> So after some fiddling around with Buildbot, it turns out that putting
> `::` in cache variable names makes it really hard to pass them on the
> command line. The cache variables have `::` replaced with `_`.
> 
> The buildbot cleanup work is ongoing in:
> 
>     https://gitlab.kitware.com/vtk/vtk/merge_requests/5039

This MR has now been merged. The buildbots are building code at least.
I'll work on cleaning up the warnings in the coming days with third
party patches and sending the relevant ones upstream.

Current state of test failures:

  - taanab and adora: CMake is warning about the inability to make
    fully correct rpath variables for targets. There may be a way to fix
    this, but it may be up to doing a suppression as well.
  - duma: it should be using OSMesa, but is not.
  - example tests: Examples aren't downloading example data properly.
  - various machines: since freetype has been updated, it seems that
    some new baselines will need to be added. I'll do that as well, but
    feel free to grab some if you want.
  - mun: static build failure with jsoncpp symbols. Will look into this
    tomorrow.
  - mun: shared+kits Python tests are failing. I suspect a missing PATH
    entry or the like. Will investgate tomorrow.

Buildbots are now building all modules allowed by their configuration.
Here is a list of modules that are explicitly disabled on machines
(usually due to lacking CUDA, Boost, or other dependency) that would
otherwise build due to the configuration[1]:

    1         'VTK_MODULE_ENABLE_VTK_GUISupportMFC:STRING': 'NO',
    1         'VTK_MODULE_ENABLE_VTK_IOOpenTurns:STRING': 'NO',
    4         'VTK_MODULE_ENABLE_VTK_RenderingExternal:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_FiltersReebGraph:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_InfovisBoostGraphAlgorithms:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_IOADIOS:STRING': 'NO',
    5         'VTK_MODULE_ENABLE_VTK_xdmf3:STRING': 'NO',
    6         'VTK_MODULE_ENABLE_VTK_IOLAS:STRING': 'NO',
    6         'VTK_MODULE_ENABLE_VTK_RenderingFreeTypeFontConfig:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_DomainsMicroscopy:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_InfovisBoost:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_IOFFMPEG:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_IOGDAL:STRING': 'NO',
    7         'VTK_MODULE_ENABLE_VTK_IOODBC:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_FiltersOpenTurns:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_IOMySQL:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_IOPDAL:STRING': 'NO',
    8         'VTK_MODULE_ENABLE_VTK_IOPostgreSQL:STRING': 'NO',
    9         'VTK_MODULE_ENABLE_VTK_RenderingOpenVR:STRING': 'NO',
    9         'VTK_MODULE_ENABLE_VTK_RenderingOptiX:STRING': 'NO',

Note that there are 9 machines, so OpenVR and OptiX are currently completely
untested on the buildbots. If these modules should be more widely
tested, feel free to open an issue for the module and which
machines/configurations should be building it and we can try and get the
dependencies installed.

Turnaround times for builds for a from-scratch build:

                configure   build   test    total   prior total
adora:          1:44        19:09  55:55    76:48   29:46
bigmac:         2:22        17:17   5:49    25:28   30:57
dejagore:       <busy with vtk-m>                   54:18
duma:           0:49        14:27  12:00    27:16   17:48
eeloo:
  +extdeps:     1:42        30:22   2:55    34:59   37:23
  +nogl:        0:49        13:45   0:13    14:47   17:54
luigi:          1:40        33:41   5:03    40:24   42:26
mun:
  (no kits):    3:15        12:38   5:38    21:31   35:15
  static:       3:20        <fails>                 39:45
  +kits:        3:19        8:29    3:55    15:43   34:55
taanab:         2:21        11:50   1:59    16:10   16:34
trey:           2:30        33:50   8:10    44:30   43:09

Note that builds are (generally) building more modules and tests than
before. From this analysis, I'm seeing that the eeloo+nogl builder has
270 tests versus the master's 807 tests, but this may be due to
dependency specifications being different. mun without kits is also
missing ~800 or so tests. adora's abysmal test time is due to timeouts
on the machine; I am seeing some X-related errors and they all seem to
be Python tests as well. dejagore is currently busy with `vtk-m`
testing, but I'm headed out for the day, so I'll leave it for tomorrow.

--Ben

[1]For example, GUISupportMFC is only explicitly disabled on Windows
machines since it "hides" on non-Windows builds. Other modules are also
masked by things like `VTK_USE_MPI`, `VTK_WRAP_PYTHON`, or platform
checks.