[vtkusers] Some VTK 5 information

[KenLists]

Here is some information on the development of VTK 5.

This email is worth paying attention to as it addresses the future of VTK
(namely VTK 5.0). We have had many discussions regarding these issues here
at Kitware and we'd like to hear what some of the other developers have to
say. Some of these changes are pretty big, many are small, but we've worked
hard to balance the needs of the future with legacy requirements from the
past. Our assumptions moving towards this major release are that we may
break some backward compatibility (mainly in the internals, little in the
public API), and that we must provide an upgrade path for users wishing to
move to VTK 5.0. Right now we are anticipating patching VTK 4.2.3 by adding
some of the features/recent additions to create VTK 4.2.4. This will serve
as the starting point for moving ahead. This will be followed by several
interim releases (at least two) which will ultimately culminate in VTK 5.0.
We expect to keep VTK 4.2 around for a long time so users can take the
necessary time to adapt to VTK5.0 as needed. This process may take a year or
more to unfold. We are very interested in feedback; this is a difficult
issue and there are likely ideas out there that we haven't thought of yet.

Before we get into the details I'd like to add some personal perspective. I
do not like the idea of breaking an API. Here at Kitware we end up spending
a lot of our time and resources rewriting books, applications, etc. and it
hurts (since we probably use VTK more than most folks). On the other hand, I
have seen the cost that failing to move forward with new ideas and excessive
attachment to backward compatibility has on software, which can and has
caused the failure of many systems. Many of the proposed changes here are
being driven by users who are dealing with large data, meaning large in
coordinate space (astrophysics, geology) or size of the data. Several of
these folks are committing significant resources towards VTK, both in terms
of funding and development. I am personally taking this very seriously and
am committed to helping the VTK community with these changes as best I can,
and doing a better job of communicating this time around. Anyway, I hope I
have your attention with this long segway :-) And you might want to grab a
beer before continuing...

Here's an itemized summary list of the changes. I will follow these with an
explanation and rationale for why and what we think the outcome will be.
I've roughly sorted the changes from smaller impact to larger.

0. Ongoing local changes

1. Implementing data arrays with a single, templated class

2. Reference counting observers

3. Higher-order cell support

4. AMR/composite dataset support

5. Use iterators to traverse collections

6. Volume rendering changes (support unstructured volume rendering)

7. Fast and slow path filter support

8. Remove old-style callbacks (i.e., SetStartMethod, etc.)

9. Blanking support for structured data

10. Pipeline execution reorganization

11. Using double rather than float in API (and internal calculations)

12. vtkMesh class


Still with me? Take another sip of beer... Here goes some explanation.


0. Ongoing local changes

Obviously there are many additions in the form of new classes, bug fixes,
documentation enhancements, etc. but these do not affect the global VTK
architecture. For example, Jean Favre has contributed a super LIC filter,
and we are on the path to develop new filters and other localized additions
for parallel processing, rendering, 3D widgets, etc.


1. Implementing data arrays with a single, templated class

Lots of data arrays, lots of duplicated code. Bad. We can make this change
with no impact on the public API....it's unlikely to affect developers
either.


2. Reference counting observers

Although vtkCommand inherits from vtkObjectBase, we do not use the reference
counting facilities. As a result we have found problems when using complex
3D widgets. This is a relatively obscure change that will not affect much of
anything.


3. Higher-order cell support

A lot of work to support (finite element) cells of arbitrary complexity.
This is an extension to VTK and should not affect anything.


4. AMR/composite dataset support

Adaptive Mesh Refinement datasets consist of hierarchies of structured data.
Many other data sets (i.e., PLOT3D files with multiple grids) form composite
structures. We belive that this functionality can be added without too much
impact since it is new, but it does depend on item #10 in order to manage
the execution of an algorithm over each of the datasets forming the
composite dataset.

Find more detail at
http://public . kitware . com/pipermail/vtk-developers/2003-October/002597 .
html


5. Use iterators to traverse collections

VTK missed the boat with iterators. It's nice to support simultaneous
iteration over a collection and to bring it more in line with modern C++.


6. Volume rendering changes (support unstructured volume rendering)

We (with Sandia Natl Lab) are adding volume rendering support for
unstructured grids. There are some changes to the volume rendering classes
(new classes, reorganization of old classes). We believe that we can do this
without impacting the existing class hierarchy.


7. Fast and slow path filter support

To a large extent we are thinking about vtkImageData here. In VTK image data
and associated filters has been implemented with pointer access and do not
handle point/cell attributes other than scalar data. However, to support
this means slowing these filters way down. So we are thinking about
reworking filters like this with support for fast and slow paths depending
on how users configure the filter. I don't see too much impact on users or
developers; maybe some derivation issues. This will likely not be completed
by VTK 5.0 and can be done as we go.


8. Remove old-style callbacks (i.e., SetStartMethod, etc.)

Command/observers good. Old style callbacks bad. Lots of obsolete code to
support the old style stuff. Time to get rid of it.


9. Blanking support for structured data

A few years ago I added blanking to the structured grid dataset. Now that we
are supporting AMR and composite data sets this functionality is becoming
more important. Moreover, we'd like to support point and cell blanking, and
methods to blank data other than just 0/1 on off arrays (e.g., specify
blanking with "all cells in this extent are off"). This will introduce minor
API changes to users and developers who used blanking (which I suspect are
few in number).


10. Pipeline execution reorganization

A bigger one. Anybody who has delved into the execution process in VTK is
probably dazed and confused. I personally suspect there are just plain wrong
implementations here and there. A big problem is the complexity that the
execution model brings to the filters and data objects. Moreover,
VTK users who are doing parallel/large data processing have been constrained
by the execution model that the current implementation enforces (e.g.,
demand driven versus data driven; how do requests for data in a particular
region or at a time step get propagated; how to manage filters that require
iterative execution or that iterate over composite data sets; how to adapt
VTK to different parallel processing models, etc.)

The goal of this work is to isolate future pipeline enhancements to the
execution model (like prioritized streaming and incremental updates).  After
the reorganization, new pipelines can be implemented externally without
changing filters and data objects. We have been working very hard on a paper
that explains our design (we'll share it with the VTK community in the near
future). We believe that we can maintain compatibility with the public API
and even with derived filters using the current pipeline mechanism. It may
be that some complex filters will require rework.


11. Using double rather than float in API (and internal calculations)

This is a severe deficiency in VTK, we need proper double support.  As you
know the current VTK API is float oriented. We have found this to be a
problem not only in filters, but also in the camera model and in widgets.
Many people cannot use VTK because their data is represented with numbers
outside the float range.  And even if the data is scaled to float range,
there are issues because a lot of operations do things like square or ^4
data, meaning the effective float range is really much smaller (1.0e9 or
less). The good news is that anywhere a float value is expected old code
will work fine, and interpreted code should not be affected much at all. It
is where C++ pointers or arrays are used that the problems set in. Note that
this change does not mean float data (vtkFloatArray) has to be stored with
doubles; it means calculations are done with double. To do this I believe
the right thing to do is to change float->double in all cases, but there are
other options like using a typedef to select float/double at compile time.
Another is to have a dual float/double interface (similar to what many of
you have already started to add).  Both these ideas are problematic because
of the complexity of carrying a dual float/double API. And using a
compile-time typedef introduces problems of its own when implementing
computation, IO, etc.


12. vtkMesh class

The idea behind this is to create a new data object vtkMesh that combines
the best qualities of both vtkPolyData and vtkUnstructuredGrid. Ultimately
we may wish to either reimplement vtkPolyData and vtkUGrid with vtkMesh, or
eliminate these classes altogether. Obviously this needs more thought for
the long term.

Motivation: there are some design problems with vtkPolyData. Namely, the use
of four arrays (verts, lines, polys, strips) and the assumed ordering of
associated cell ids causes problems (inserting a polygon first and then a
line wreaks havoc because the traversal later on assumes lower order
primitives first effectively changing the cell ids). Also, vtkPolyData and
vtkUGrid are essentially the same thing, do we really have to have two
representations?  We also have discovered ways to reduce the total storage,
improve neighborhood operation speed, etc. and otherwise improve the
performance of the classes.

So here is the plan for handling VTK5 based on feedback we have received so
far. Since the transition to VTK5 will contain significant changes we are
creating a separate repository for VTK4 so that people who want to move
forward with VTK4 can do so. We have created VTK4 and VTK4Data CVS
repositories as:

cvs -d :pserver:yourname at www.vtk.org:/cvsroot/VTK co VTK4

cvs -d :pserver:yourname at www.vtk.org:/cvsroot/VTK co VTK4Data

These repositories are starting from the VTK 4.2.4 version of VTK and are
designed to make it easy for people to continue using VTK version 4 into the
future. Between version 4.2.3 and version 4.2.4 we have added some of the
new functionality that is in the CVS version of VTK. This VTK4 tree will be
like a CVS version of VTK4.2 that people can do development on (Kitware
folks will stay out of this tree). Feel free to try out the VTK4 repository
and offer comments. 

The main CVS tree for VTK will be where the VTK5 development takes place. It
will be done in stages with intermediate releases such as 4.4, 4.6, 4.8,
which will mainly be dividing points between key changes to help people
incrementally move to VTK 5.0.

The release-4-2 branch of the main VTK tree will also continue to exist as
the formal VTK 4.2 branch on which generally only bug fixes will be made. I
do not anticipate anyone but myself committing changes to the release-4-2
branch of VTK. As additional bug fixes happen there may be a 4.2.5 release
etc. I do not anticipate keeping the VTK4 tree and the VTK 4.2 branch in
sync. Currently my view is that the VTK4 tree includes the option for people
to continue developing new classes etc. The vtk 4.2 branch is really only
for bug fixes so the two trees will likely diverge. 

I imagine that some of the existing VTK dashboards will become VTK4
dashboards so that we can have some continuing testing on that source tree.
My guess for now is that they will show up on the main VTK dashboard much
like the VTK 4.2 branch dashboards show up but a separate dashboard could be
created if people feel that is the way to go.

So in summary

1) VTK 5 soon to be developed on the main tree
2) VTK 4.2 release continues to be a branch on the main tree
3) VTK4 tree supports future development and expansion of VTK 4.2 for those
who don't want to go to VTK 5 but cannot stick with the 4.2 release  

We will have snapshots, but they will not be full releases. The FAQ etc will
still point to VTK 4.2 as the release but at least the snapshots will
provide points for people who need a recent but stable version of VTK. A
rough roadmap (I'll update the FAQ) is attached below:

VTK 4.4 (intermediate release, end of year)
  - convert APIs to double
  - remove old callbacks (mostly done but not checked in)
  - blanking (done I think)
  - ref count observers (done)
  - switch collections to use iterators
  - improve copyright (done)

VTK 4.8 (intermediate release, end of February)
  - create mesh class to combine polydata with ugrid
  - volume rendering changes (maybe in 4.4)

VTK 5.0 (major release, end of June)
  - new pipeline mechanism
  - time support
  - true AMR support

Thanks
Ken