[vtk-developers] Pay attention, grab a beer: VTK 5.0

[Jeff Lee]

Hi Developers,

It is clear that some classes will need to be re-designed to accomplish 
the proposed features.  We all hate API changes because of the work it 
requires, but I am in favor of API changes if they are justified.  Good 
software evolves.  Although some of the changes are real monsters, I 
think they all have merit, and a couple are absolutely essential.

The changes closest to my heart would be data-driven pipeline execution, 
vtkMesh, iterators.  I would like to see stl/vtkstl allowed in the api.  
I would like to see a better way of registering new cells into vtk.  I 
am also looking forward to some sort of data-adapter that makes it 
easier to wrap existing memory for use in vtk.

I look forward to more details on the proposed changes.

-Jeff

-- 
Jeff Lee
Senior Software Engineer
Computational Dynamics North America Ltd
21 Lafayette Street, Suite 230
Lebanon NH 03766 USA
fax:   603 643 9994
phone: 603 643 9993 x109
http://www.cd-adapco.com


On 10/31/03 1:05 PM, "Will Schroeder" <will.schroeder at kitware.com> wrote:

>> Hi Folks-
>> 
>> This email is worth paying attention to as it addresses the future of VTK
>> (namely VTK 5.0). We have had many discussions regarding these issues here at
>> Kitware and we'd like to hear what some of the other developers have to
>> say. Some of these changes are pretty big, many are small, but we've worked
>> hard to balance the needs of the future with legacy requirements from the
>> past. Our assumptions moving towards this major release are that we may break
>> some backward compatibility (mainly in the internals, little in the public
>> API), and that we must provide an upgrade path for users wishing to move to
>> VTK 5.0. Right now we are anticipating patching VTK 4.2.3 by adding some of
>> the features/recent additions to create VTK 4.2.4. This will serve as the
>> starting point for moving ahead. This will be followed by several interim
>> releases (at least two) which will ultimately culminate in VTK 5.0. We expect
>> to keep VTK 4.2 around for a long time so users can take the necessary time
>> to adapt to VTK5.0 as needed. This process may take a year or more to unfold.
>> We are very interested in feedback; this is a difficult issue and there are
>> likely ideas out there that we haven't thought of yet.
>> 
>> Before we get into the details I'd like to add some personal perspective. I
>> do not like the idea of breaking an API. Here at Kitware we end up spending a
>> lot of our time and resources rewriting books, applications, etc. and it
>> hurts (since we probably use VTK more than most folks). On the other hand, I
>> have seen the cost that failing to move forward with new ideas and excessive
>> attachment to backward compatibility has on software, which can and has
>> caused the failure of many systems. Many of the proposed changes here are
>> being driven by users who are dealing with large data, meaning large in
>> coordinate space (astrophysics, geology) or size of the data. Several of
>> these folks are committing significant resources towards VTK, both in terms
>> of funding and development. I am personally taking this very seriously and am
>> committed to helping the VTK community with these changes as best I can, and
>> doing a better job of communicating this time around. Anyway, I hope I have
>> your attention with this long segway  And you might want to grab a beer
>> before continuing...
>> 
>> Here's an itemized summary list of the changes. I will follow these with an
>> explanation and rationale for why and what we think the outcome will be. I've
>> roughly sorted the changes from smaller impact to larger.
>> 
>> 0. Ongoing local changes
>> 
>> 1. Implementing data arrays with a single, templated class
>> 
>> 2. Reference counting observers
>> 
>> 3. Higher-order cell support
>> 
>> 4. AMR/composite dataset support
>> 
>> 5. Use iterators to traverse collections
>> 
>> 6. Volume rendering changes (support unstructured volume rendering)
>> 
>> 7. Fast and slow path filter support
>> 
>> 8. Remove old-style callbacks (i.e., SetStartMethod, etc.)
>> 
>> 9. Blanking support for structured data
>> 
>> 10. Pipeline execution reorganization
>> 
>> 11. Using double rather than float in API (and internal calculations)
>> 
>> 12. vtkMesh class
>> 
>> 
>> Still with me? Take another sip of beer... Here goes some explanation.
>> 
>> 
>> 0. Ongoing local changes
>> 
>> Obviously there are many additions in the form of new classes, bug fixes,
>> documentation enhancements, etc. but these do not affect the global VTK
>> architecture. For example, Jean Favre has contributed a super LIC filter, and
>> we are on the path to develop new filters and other localized additions for
>> parallel processing, rendering, 3D widgets, etc.
>> 
>> 
>> 1. Implementing data arrays with a single, templated class
>> 
>> Lots of data arrays, lots of duplicated code. Bad. We can make this change
>> with no impact on the public API....it's unlikely to affect developers either.
>> 
>> 
>> 2. Reference counting observers
>> 
>> Although vtkCommand inherits from vtkObjectBase, we do not use the reference
>> counting facilities. As a result we have found problems when using complex 3D
>> widgets. This is a relatively obscure change that will not affect much of
>> anything.
>> 
>> 
>> 3. Higher-order cell support
>> 
>> A lot of work to support (finite element) cells of arbitrary complexity. This
>> is an extension to VTK and should not affect anything.
>> 
>> 
>> 4. AMR/composite dataset support
>> 
>> Adaptive Mesh Refinement datasets consist of hierarchies of structured data.
>> Many other data sets (i.e., PLOT3D files with multiple grids) form composite
>> structures. We belive that this functionality can be added without too much
>> impact since it is new, but it does depend on item #10 in order to manage the
>> execution of an algorithm over each of the datasets forming the composite
>> dataset.
>> 
>> Find more detail at
>> http://public.kitware.com/pipermail/vtk-developers/2003-October/002597.html
>> 
>> 
>> 5. Use iterators to traverse collections
>> 
>> VTK missed the boat with iterators. It's nice to support simultaneous
>> iteration over a collection and to bring it more in line with modern C++.
>> 
>> 
>> 6. Volume rendering changes (support unstructured volume rendering)
>> 
>> We (with Sandia Natl Lab) are adding volume rendering support for unstructured
>> grids. There are some changes to the volume rendering classes (new classes,
>> reorganization of old classes). We believe that we can do this without
>> impacting the existing class hierarchy.
>> 
>> 
>> 7. Fast and slow path filter support
>> 
>> To a large extent we are thinking about vtkImageData here. In VTK image data
>> and associated filters has been implemented with pointer access and do not
>> handle point/cell attributes other than scalar data. However, to support this
>> means slowing these filters way down. So we are thinking about reworking
>> filters like this with support for fast and slow paths depending on how
>> users configure the filter. I don't see too much impact on users or
>> developers; maybe some derivation issues. This will likely not be completed
>> by VTK 5.0 and can be done as we go.
>> 
>> 
>> 8. Remove old-style callbacks (i.e., SetStartMethod, etc.)
>> 
>> Command/observers good. Old style callbacks bad. Lots of obsolete code
>> to support the old style stuff. Time to get rid of it.
>> 
>> 
>> 9. Blanking support for structured data
>> 
>> A few years ago I added blanking to the structured grid dataset. Now that we
>> are supporting AMR and composite data sets this functionality is becoming
>> more important. Moreover, we'd like to support point and cell blanking, and
>> methods to blank data other than just 0/1 on off arrays (e.g., specify
>> blanking with "all cells in this extent are off"). This will introduce minor
>> API changes to users and developers who used blanking (which I suspect are
>> few in number).
>> 
>> 
>> 10. Pipeline execution reorganization
>> 
>> A bigger one. Anybody who has delved into the execution process in VTK is
>> probably dazed and confused. I personally suspect there are just plain
>> wrong implementations here and there. A big problem is the complexity that
>> the execution model brings to the filters and data objects. Moreover,
>> VTK users who are doing parallel/large data processing have been constrained
>> by the execution model that the current implementation enforces (e.g.,
>> demand driven versus data driven; how do requests for data in a particular
>> region or at a time step get propagated; how to manage filters that
>> require iterative execution or that iterate over composite data sets;
>> how to adapt VTK to different parallel processing models, etc.)
>> 
>> The goal of this work is to isolate future pipeline enhancements to the
>> execution model (like prioritized streaming and incremental updates).  After
>> the reorganization, new pipelines can be implemented externally without
>> changing filters and data objects. We have been working very hard on a paper
>> that explains our design (we'll share it with the VTK community in the near
>> future). We believe that we can maintain compatibility with the public API
>> and even with derived filters using the current pipeline mechanism. It may be
>> that some complex filters will require rework.
>> 
>> 
>> 11. Using double rather than float in API (and internal calculations)
>> 
>> This is a severe deficiency in VTK, we need proper double support.  As you
>> know the current VTK API is float oriented. We have found this to be a
>> problem not only in filters, but also in the camera model and in
>> widgets. Many people cannot use VTK because their data is represented with
>> numbers outside the float range.  And even if the data is scaled to float
>> range, there are issues because a lot of operations do things like square or
>> ^4 data, meaning the effective float range is really much smaller. The good
>> news is that anywhere a float value is expected old code will work fine, and
>> interpreted code should not be affected much at all. It is where C++ pointers
>> or arrays are used that the problems set in. Note that this change does not
>> mean float data has to be represented with doubles; it means calculations are
>> done with double. To do this I believe the right thing to do is to change
>> float->double in all cases, but there are other options like using a typedef
>> to select float/double at compile time. Another is to have a dual float/double
>> interface (similar to what many of you have already started to add).  Both
>> these ideas are problematic because of the complexity of carrying a dual
>> float/double API. And using a compile-time typedef introduces problems of its
>> own when implementing computation, IO, etc.
>> 
>> 
>> 12. vtkMesh class
>> 
>> The idea behind this is to create a new data object vtkMesh that combines the
>> best qualities of both vtkPolyData and vtkUnstructuredGrid. Ultimately we may
>> wish to either reimplement vtkPolyData and vtkUGrid with vtkMesh, or
>> eliminate these classes altogether. Obviously this needs more thought for the
>> long term.
>> 
>> Motivation: there are some design problems with vtkPolyData. Namely, the use
>> of four arrays (verts, lines, polys, strips) and the assumed ordering of
>> associated cell ids causes problems (inserting a polygon first and then a
>> line wreaks havoc because the traversal later on assumes lower order
>> primitives first effectively changing the cell ids). Also, vtkPolyData and
>> vtkUGrid are essentially the same thing, do we really have to have two
>> representations?  We also have discovered ways to reduce the total storage,
>> improve neighborhood operation speed, etc. and otherwise improve the
>> performance of the classes.
>> 
>> 
>> 
>> Okay, if you're still here I'd like to reiterate that we are going to do our
>> best to minimize the impact on both developers and users. And if change is
>> necessary we will communicate and help provide a way to migrate to the newer
>> code base. Comments?
>> 
>> 
>> _______________________________________________
>> vtk-developers mailing list
>> vtk-developers at vtk.org
>> http://www.vtk.org/mailman/listinfo/vtk-developers
>  
>