[vtk-developers] [BUILDBOT] New buildbot features!

[Ben Boeckel]

Hi,

As I see some people have noticed, buildbot can now be driven by `Do:
test` which triggers ~immediately rather than on a 10 minute schedule.
With this come some other nifty bits :) .

You may also have noticed buildbot now comments back to your MR when it
is complete :) . If anyone sees duplicate messages where the first does
not involve "slave lost? will retry in the future" somewhere in the
list, please comment while mentioning @buildbot.

First, some important information:

If you are currently using `@buildbot test`, if `Do: test` is found, any
`@buildbot test` mechanism is ignored, so moving to `Do: test` will not
cause duplicate builds. A transistion of a few weeks should suffice
before we turn off `@buildbot test`. It also needs to be at the bottom
of your comment, the same as other kwrobot commands.

It is currently only on VTK for now; I'll announce to the other projects
when it has been enabled for them (kwrobot will tell you if it is not
ready as well); you will probably then need to edit your comment to
remove `Do: test` so `@buildbot test` doesn't skip it though.

And now for the fun bits:

With the new `Do: test` command, buildbot keeps a list of active
commands for a merge request. For example, this may be used to build
only on Python3 builders:

    Do: test --regex-include \+python3

to *add* MPI builders to the set, give this command:

    Do: test --regex-include \+mpi

When these two commands are active, any builder with +python3 or +mpi in
its name will be built for updates to the MR. If, however, you now only
need MPI builders, this command:

    Do: test --clear --regex-include \+mpi

will empty out the command list before adding the command for +mpi.

To test any Python except on Windows with MPI:

    Do: test --regex-include \+python --regex-exclude -windows.*\+mpi

On any branch update, all active commands will be used, but --oneshot
may be used to not remember the command.

The breakdown for build names is as follows:

    project-host-os-libtype-buildtype+feature1+feature2

  * project: always "vtk" for vtk
  * host: the buildbot host
  * os: one of "windows", "osx", or "linux"
  * libtype: "shared" or "static"
  * buildtype: "release" or "debug"

And a grouped list of features (not exhaustive):

  * python
  * python3
  * tcl
  * java

  * opengl2 (OpenGL1 if not there; will be flipped to be +opengl1 at
    some point)

  * mpi
  * qt
  * qt5

  * clang
  * gcc (this is not used everywhere yet: if not specified and not
    Windows, it's probably gcc; this will be fixed when we rename
    builders to turn +opengl2 -> +opengl1)
  * vs (Visual Studio generator; all Windows builders are MSVC)
  * icc
  * pgi

  * tbb
  * openmp

All features in a build name are in alphabetical order, so matching a
collection of features shouldn't need too many regex hacks.

Some nitty-gritty details should the need arise:

Arguments may not contain spaces (there should be no reason to) and
there is no escaping other than Python's regex nor quoting (i.e., the
command is split on spaces and that's it). Extra spaces between
arguments shouldn't be a problem.

The command is applied in whitelist -> blacklist order, so
--regex-exclude will never be overridden within each command; however,
each command is considered separately and then union'd with the other
command results (so that you can include builders later and not have to
hunt down the command excluding it from before).

Any errors in argument parsing renders the entire command as ignored.

The full list of arguments:

  * --oneshot
        only build the *current* hash of the branch; updates will not be
        built using this command
  * --stop
        clear the list of commands for the merge request
  * --superbuild
        build the superbuilds related to the project (vtk-superbuild is
        currently excluded from this since it is only a nightly test)
  * --clear:
        clear previous commands and add this command to the list
  * --regex-include <arg>
        only build on builders matching <arg> (a Python regex)
  * --regex-exclude <arg>
        excludes builds on builders matching <arg> (a Python regex)

--Ben