VTK/Remove BTX ETX

The new wrappers, introduced in VTK 5.8, do not use the BTX/ETX markers for in-tree VTK wrapper generation. In the past, these markers were primarily used to mark portions of the code that vtkParse was unable to parse. However, the new wrappers are able to understand much more of the C++ grammar, so the use of these markers for that purpose has become largely unnecessary.

The secondary use of BTX/ETX markers was to exclude methods with parameters that were unwrapped object types. Without the BTX/ETX markers, the wrappers would generate wrapper code for these parameters on the assumption that these parameters were VTK objects (i.e. derived from vtkObjectBase), and that code would then fail to compile when the objects lacked certain vtkObjectBase methods such as New(), Register(), and Delete().

For in-tree VTK wrapper generation, the new wrappers can automatically determine whether any encountered type is derived from vtkObjectBase, which makes the BTX/ETX markers unnecessary. Therefore, for in-tree wrapper builds, the new wrappers ignore BTX/ETX markers completely. For out-of-tree wrapper generation, e.g. generation of wrappers for external packages, type hierarchies cannot be automatically deduced and BTX/ETX markers are used as they were previously. More information on this is given below.

Replacement for BTX/ETX

There is a new preprocessor symbol, __WRAP__, which is defined by vtkParse. It can be used to mark blocks of code that are to be skipped by the parser. It should be used sparingly, any need for its use should be considered to be due to inadequacy in vtkParse.

#ifndef __WRAP__
  // code to be skipped by wrappers
#endif

To keep wrapping behavior consistent between vtkParse and Mummy, it might be useful to define the following:

#define __WRAP__ __GCCXML__

Use of BTX/ETX in third-party projects

The behavior of vtkParse with respect to BTX/ETX is different for external projects as compared to code within VTK itself. In external projects, BTX/ETX function exactly as they did before the new wrappers were introduced. This is for the sake of backwards compatibility.

Ideally, external projects should also remove all use of BTX/ETX, but unfortunately they cannot do this yet. The reason is that, in order to wrap methods that take arguments that are not simple known types like int, float, etcetera, it is necessary that vtkParse can do "wrap-time type identification" on those types. And for that, it needs to use the vtkKitHierarchy.txt files that were introduced in VTK 5.8. Here is an example:

void vtkSomeClass::SetVariantValue(MyIndexType idx, vtkVariant val);

The vtkCommonHierarchy.txt file contains an entry for vtkVariant that shows its pedigree, i.e. so that the wrappers know that it is a wrapped type but is not derived from vtkObjectBase. Likewise, one of the other hierarchy.txt files might have a typedef entry for MyIndexType to let the wrappers know that it is "unsigned long". The wrappers use the hierarchy.txt files to elucidate type information, so that they do not have to parse all the files included by the .h files they are wrapping.

The solution: How to completely remove BTX/ETX

Work is underway to allow the wrappers to pull in the .h files that are included by all wrapped .h files. In fact this is already done in part, because the included .h files are run through an in-line preprocessor in order to extract any #define statements that might generate macros that are needed within the wrapped .h file. Preprocessing alone, however, is not sufficient for our needs. In fact three pieces are needed in order to allow third-party packages to execute the wrapper tools after all BTX/ETX markers have been removed:

1. All .h files included by the wrapped header file must be parsed and their contents temporarily stored by the wrapper tools. Note that these included headers will only be parsed, not wrapped.
2. A search routine must be added to allow the wrappers to search through the parsed data for all the included files for any types that are encountered in any wrapped class methods.
3. If the search for an encountered type fails, but a forward declaration for that type exists, then the wrappers must attempt to load the header file for that type. For example, if the type vtkLookupTable is encountered then vtkLookupTable.h will be loaded if it exists.

If the header file for any parameter type cannot be found (e.g. if the header file is not named after the type), then the methods that use that type will not be wrapped. Effectively, this will provide the wrappers with the means of automatically excluding methods from being wrapped, without the need for programmers to use BTX/ETX markers to exclude these methods.

Progress

The new wrappers have been able to search for and load header files since at least July 2010, so item (3) is mostly complete.

Parsing of included header files (1) was added to the wrapper-parser-enhancements topic branch on April 3, 2012.

Searching through included headers for types (2) is a work in progress as of April 16, 2012. It is expected to be completed sometime within the next two months.

After this work is complete, all BTX/ETX markers can be safely removed from the VTK code, both within VTK itself and within all external packages that derive from VTK classes.

New Progress

3.5 years have passed since the last progress report, so how about an update!

Items (1) and (2) were never done. However, the vtkModuleHierarchy.txt files are exported for use by third-party packages, and they provide all of the type information that (1) and (2) would have provided.

So, it is reasonable to remove the BTX/ETX markers now. In fact, it would have been reasonable to do so for the release of VTK 6.0. Better late than never!