VTK/Wrapper Update 2012

The VTK wrappers received a major overhaul in 2010 prior to the release of VTK 5.8. Work on improving the wrappers has continued (though at a slower pace) and this document aims to describe the wrapper goals for VTK 6.0.

Overview

The "Wrapper Overhaul 2010" project (Source article) had four primary goals, three of which were completed:

1. Cleaning up the wrapper code (always more to do, though...)
2. Wrapping vtkStdString (and vtkUnicodeString)
3. Wrapping vtkVariant (and other non-vtkObject types)
4. Eliminating BTX/ETX from the VTK header files

Item 3 was eventually extended to cover templated classes like vtkVector, which is a good demonstration of the power of the VTK wrapping tools. Items 3 and 4 necessitated the adoption of the vtkWrapHierarchy tool, which is only now (Aug 2012) being adopted into VTK-based projects like ParaView. As a result, Item 4 was not completed... it is possible to build VTK with all the BTX/ETX markers removed, but this will cause external wrappers like those in ParaView to fail.

The new goals, which are targeted for the VTK 6.0 release, are:

1. More wrapper code cleanup (DONE, but always more to do here)
2. Integrate wrapping with the new modular VTK (DONE)
3. Make the wrapping tools themselves into a VTK module (DONE)
4. Finish the work on removing BTX/ETX

Detailed Descriptions

Code Cleanup and Enhancement

Nearly all the parsing code for the wrappers was contained in one huge file, vtkParse.y. In 2010 the code in this file was cleaned up, but the file itself was still huge. Now, the parser code has been reorganized into several smaller "vtkParse..." files such as:

• vtkParseMain.c - argument parsing and other useful things for wrapper tool executables
• vtkParseHierarchy.c - allow wrapper tools to introspect the class hierarchy (created in 2010)
• vtkParseData.c - data structures for describing C++ classes, constants, functions, templates, and types
• vtkParseString.c - an efficient replacement for the dreaded "strdup()" that allows memory reclamation
• vtkParseMangle.c - provide symbol mangling utilities
• vtkParseExtras.c - a catch-all for utilities that don't fit elsewhere (created in 2010)

The tokenizer file, vtkParse.l, was much smaller than vtkParse.y but it has also been reorganized and supplemented:

• vtkParsePreprocess.c - preprocess C++ files, keep a table of macro definitions

The fact that the wrapper tools can expand macros and follow preprocessor directives makes them much more robust, though in order for the full power of the preprocessor to be realized the wrapper tools must be supplied with a list of include directories given on the command line as "-I" arguments.

Along with the code cleanup, the grammar rules in vtkParse.y have been rewritten. This started with trial-and-error modifications to enable the parser to wrap header files that contained ITK class declarations, and culminated in the renaming and rewriting the rules to approximate the grammar for the 1998 ISO C++ standard. With these changes, it is hoped that only truly pathological code will be able to break the parser without also breaking at least one of the supported VTK compilers.

Make the Wrapper Tools work with Modular VTK

Most of the necessary work was done as part of the rewriting of the VTK cmake scripts during the VTK modularization project. A new set of high-level wrapping macros now exist that are automatically called for each module that exists (that is, except for the modules that are excluded from wrapping).

One problem that arose from modularization is that the large number of directories (as compared to pre-modularized VTK) resulted in very long include directory search paths. Because the include paths are passed to the wrapper tools on the command line, this resulted in a command lines that were too long for the Windows CMD shell. To overcome this limitation, a response file option was added to the wrapper tool executables, to allow some or all of the command line arguments to be passed in a file. The idea for the response was borrowed from MSVC and gcc, which both have the same feature (with the same syntax).

The three wrapping directories Wrapping/Python, Wrapping/Tcl, and Wrapping/Java were themselves converted into modules, but they are very different from other VTK modules. For example, the bulk of the python wrapping process is performed by the wrapping macros in the VTK/CMake/vtkWrapping.cmake while each VTK module is built. It does not occur while the vtkWrappingPython module itself is being built, the vtkWrappingPython module is only responsible for building a utility library, a vtkpython executable, and a vtk-python package. The VTK wrapping process is, and probably always will be, distributed.

Making a WrapperTools Module

In order to build vtkWrapCientServer, the ParaView cmake files must grab certain wrapper source files from the VTK source code and compile them. This means that any cleanup or reorganization of the VTK wrapper tool code necessitates manual adjustments to the build scripts for vtkWrapClientServer. If the VTK wrapper tools existed as a library module, then it would be much easier to create and maintain external wrapper tools like vtkWrapClientServer.

Remove BTX/ETX

Completing the removal of BTX/ETX from the VTK header files requires that external projects use vtkWrapHierarchy to provide a "bird's eye view" of all types provided by the project. If external projects use the VTK module macros to build their own libraries and executables, then this will occur automatically, as long as VTK itself has properly exported its type information in the form of hierarchy files.

Wish List

These items are unlikely to be completed for VTK 6.0, but sure would be nice to have eventually:

Eliminate Hierarchy Requirement

Now that the wrapper tools are provided with "-I" arguments so that they can search for included files, the vtkWrapHierarchy tool is not strictly necessary. However, further work has to be done before vtkWrapHierarchy (and the headaches that come with it) can be eliminated completely. This "further work" involves writing code for the parser that performs context-sensitive name lookups, which can be very complicated e.g. for templated code.

Support Doxygen-Style Comments

The Tcl and Python wrappers include full copies of the VTK header-file documentation. Many users rely on this documentation extensively. Currently, the wrapper tools require all documentation to be in the "//Description" format used by VTK. It would be very nice to also support doxygen-style comments in the header files, rather than to continue to require a format that is used only by VTK.

Python 3 Support

Python 3 introduced major API changes for strings and ints, and minor API changes elsewhere.

• It will be possible to support Python 2.3 though Python 3
• "Classic" classes are gone, PyVTKObject and PyVTKClass will need to be replaced
• PyIntObject and PyStringObject are absent in Python 3, but are used widely in VTK/Python
• Python 3 uses unicode for all strings (ramifications for VTK?)
• Language changes: some examples will have to be rewritten to work with python 3