[vtk-developers] Pay attention, grab a beer: VTK 5.0

[Mark Beall]

Will,
I'll make a general comment on the proposed changes, specific ones I'll
direct to you later.

The reality is that developing a complex piece of software like VTK means
that you will occasionally make bad choices that aren't obvious until later.
You can choose to forever live with those decisions and their consequences,
even though they cause you problems everyday, or you can decide to fix them.
Fixing them, in the short term, causes another set of problems, but in the
long term allows growth. Not fixing them will allow them to gnaw at you
forever with no relief possible. Pretty much like life in general.

I say, make the changes necessary to keep VTK healthy and keep doing this as
necessary in the future. Certainly there will be people who will grumble
about such things. They have the option of recognizing the value that they
get from VTK and Kitware's efforts on it, or they can simply branch VTK and
live in their own little world where they don't get those benefits in the
future. That's their choice.

mark


Mark W. Beall, Ph.D.                           mbeall at simmetrix.com
President                                      (518) 348-1639 x101
Simmetrix, Inc.                                www.simmetrix.com
10 Halfmoon Executive Park Drive
Clifton Park, NY  12065



On 10/31/03 1:05 PM, "Will Schroeder" <will.schroeder at kitware.com> wrote:

> Hi Folks-
> 
> This email is worth paying attention to as it addresses the future of VTK
> (namely VTK 5.0). We have had many discussions regarding these issues here at
> Kitware and we'd like to hear what some of the other developers have to
> say. Some of these changes are pretty big, many are small, but we've worked
> hard to balance the needs of the future with legacy requirements from the
> past. Our assumptions moving towards this major release are that we may break
> some backward compatibility (mainly in the internals, little in the public
> API), and that we must provide an upgrade path for users wishing to move to
> VTK 5.0. Right now we are anticipating patching VTK 4.2.3 by adding some of
> the features/recent additions to create VTK 4.2.4. This will serve as the
> starting point for moving ahead. This will be followed by several interim
> releases (at least two) which will ultimately culminate in VTK 5.0. We expect
> to keep VTK 4.2 around for a long time so users can take the necessary time
> to adapt to VTK5.0 as needed. This process may take a year or more to unfold.
> We are very interested in feedback; this is a difficult issue and there are
> likely ideas out there that we haven't thought of yet.
> 
> Before we get into the details I'd like to add some personal perspective. I
> do not like the idea of breaking an API. Here at Kitware we end up spending a
> lot of our time and resources rewriting books, applications, etc. and it
> hurts (since we probably use VTK more than most folks). On the other hand, I
> have seen the cost that failing to move forward with new ideas and excessive
> attachment to backward compatibility has on software, which can and has
> caused the failure of many systems. Many of the proposed changes here are
> being driven by users who are dealing with large data, meaning large in
> coordinate space (astrophysics, geology) or size of the data. Several of
> these folks are committing significant resources towards VTK, both in terms
> of funding and development. I am personally taking this very seriously and am
> committed to helping the VTK community with these changes as best I can, and
> doing a better job of communicating this time around. Anyway, I hope I have
> your attention with this long segway :-) And you might want to grab a beer
> before continuing...
> 
> Here's an itemized summary list of the changes. I will follow these with an
> explanation and rationale for why and what we think the outcome will be. I've
> roughly sorted the changes from smaller impact to larger.
> 
> 0. Ongoing local changes
> 
> 1. Implementing data arrays with a single, templated class
> 
> 2. Reference counting observers
> 
> 3. Higher-order cell support
> 
> 4. AMR/composite dataset support
> 
> 5. Use iterators to traverse collections
> 
> 6. Volume rendering changes (support unstructured volume rendering)
> 
> 7. Fast and slow path filter support
> 
> 8. Remove old-style callbacks (i.e., SetStartMethod, etc.)
> 
> 9. Blanking support for structured data
> 
> 10. Pipeline execution reorganization
> 
> 11. Using double rather than float in API (and internal calculations)
> 
> 12. vtkMesh class
> 
> 
> Still with me? Take another sip of beer... Here goes some explanation.
> 
> 
> 0. Ongoing local changes
> 
> Obviously there are many additions in the form of new classes, bug fixes,
> documentation enhancements, etc. but these do not affect the global VTK
> architecture. For example, Jean Favre has contributed a super LIC filter, and
> we are on the path to develop new filters and other localized additions for
> parallel processing, rendering, 3D widgets, etc.
> 
> 
> 1. Implementing data arrays with a single, templated class
> 
> Lots of data arrays, lots of duplicated code. Bad. We can make this change
> with no impact on the public API....it's unlikely to affect developers either.
> 
> 
> 2. Reference counting observers
> 
> Although vtkCommand inherits from vtkObjectBase, we do not use the reference
> counting facilities. As a result we have found problems when using complex 3D
> widgets. This is a relatively obscure change that will not affect much of
> anything.
> 
> 
> 3. Higher-order cell support
> 
> A lot of work to support (finite element) cells of arbitrary complexity. This
> is an extension to VTK and should not affect anything.
> 
> 
> 4. AMR/composite dataset support
> 
> Adaptive Mesh Refinement datasets consist of hierarchies of structured data.
> Many other data sets (i.e., PLOT3D files with multiple grids) form composite
> structures. We belive that this functionality can be added without too much
> impact since it is new, but it does depend on item #10 in order to manage the
> execution of an algorithm over each of the datasets forming the composite
> dataset.
> 
> Find more detail at
> http://public.kitware.com/pipermail/vtk-developers/2003-October/002597.html
> 
> 
> 5. Use iterators to traverse collections
> 
> VTK missed the boat with iterators. It's nice to support simultaneous
> iteration over a collection and to bring it more in line with modern C++.
> 
> 
> 6. Volume rendering changes (support unstructured volume rendering)
> 
> We (with Sandia Natl Lab) are adding volume rendering support for unstructured
> grids. There are some changes to the volume rendering classes (new classes,
> reorganization of old classes). We believe that we can do this without
> impacting the existing class hierarchy.
> 
> 
> 7. Fast and slow path filter support
> 
> To a large extent we are thinking about vtkImageData here. In VTK image data
> and associated filters has been implemented with pointer access and do not
> handle point/cell attributes other than scalar data. However, to support this
> means slowing these filters way down. So we are thinking about reworking
> filters like this with support for fast and slow paths depending on how
> users configure the filter. I don't see too much impact on users or
> developers; maybe some derivation issues. This will likely not be completed
> by VTK 5.0 and can be done as we go.
> 
> 
> 8. Remove old-style callbacks (i.e., SetStartMethod, etc.)
> 
> Command/observers good. Old style callbacks bad. Lots of obsolete code
> to support the old style stuff. Time to get rid of it.
> 
> 
> 9. Blanking support for structured data
> 
> A few years ago I added blanking to the structured grid dataset. Now that we
> are supporting AMR and composite data sets this functionality is becoming
> more important. Moreover, we'd like to support point and cell blanking, and
> methods to blank data other than just 0/1 on off arrays (e.g., specify
> blanking with "all cells in this extent are off"). This will introduce minor
> API changes to users and developers who used blanking (which I suspect are
> few in number).
> 
> 
> 10. Pipeline execution reorganization
> 
> A bigger one. Anybody who has delved into the execution process in VTK is
> probably dazed and confused. I personally suspect there are just plain
> wrong implementations here and there. A big problem is the complexity that
> the execution model brings to the filters and data objects. Moreover,
> VTK users who are doing parallel/large data processing have been constrained
> by the execution model that the current implementation enforces (e.g.,
> demand driven versus data driven; how do requests for data in a particular
> region or at a time step get propagated; how to manage filters that
> require iterative execution or that iterate over composite data sets;
> how to adapt VTK to different parallel processing models, etc.)
> 
> The goal of this work is to isolate future pipeline enhancements to the
> execution model (like prioritized streaming and incremental updates).  After
> the reorganization, new pipelines can be implemented externally without
> changing filters and data objects. We have been working very hard on a paper
> that explains our design (we'll share it with the VTK community in the near
> future). We believe that we can maintain compatibility with the public API
> and even with derived filters using the current pipeline mechanism. It may be
> that some complex filters will require rework.
> 
> 
> 11. Using double rather than float in API (and internal calculations)
> 
> This is a severe deficiency in VTK, we need proper double support.  As you
> know the current VTK API is float oriented. We have found this to be a
> problem not only in filters, but also in the camera model and in
> widgets. Many people cannot use VTK because their data is represented with
> numbers outside the float range.  And even if the data is scaled to float
> range, there are issues because a lot of operations do things like square or
> ^4 data, meaning the effective float range is really much smaller. The good
> news is that anywhere a float value is expected old code will work fine, and
> interpreted code should not be affected much at all. It is where C++ pointers
> or arrays are used that the problems set in. Note that this change does not
> mean float data has to be represented with doubles; it means calculations are
> done with double. To do this I believe the right thing to do is to change
> float->double in all cases, but there are other options like using a typedef
> to select float/double at compile time. Another is to have a dual float/double
> interface (similar to what many of you have already started to add).  Both
> these ideas are problematic because of the complexity of carrying a dual
> float/double API. And using a compile-time typedef introduces problems of its
> own when implementing computation, IO, etc.
> 
> 
> 12. vtkMesh class
> 
> The idea behind this is to create a new data object vtkMesh that combines the
> best qualities of both vtkPolyData and vtkUnstructuredGrid. Ultimately we may
> wish to either reimplement vtkPolyData and vtkUGrid with vtkMesh, or
> eliminate these classes altogether. Obviously this needs more thought for the
> long term.
> 
> Motivation: there are some design problems with vtkPolyData. Namely, the use
> of four arrays (verts, lines, polys, strips) and the assumed ordering of
> associated cell ids causes problems (inserting a polygon first and then a
> line wreaks havoc because the traversal later on assumes lower order
> primitives first effectively changing the cell ids). Also, vtkPolyData and
> vtkUGrid are essentially the same thing, do we really have to have two
> representations?  We also have discovered ways to reduce the total storage,
> improve neighborhood operation speed, etc. and otherwise improve the
> performance of the classes.
> 
> 
> 
> Okay, if you're still here I'd like to reiterate that we are going to do our
> best to minimize the impact on both developers and users. And if change is
> necessary we will communicate and help provide a way to migrate to the newer
> code base. Comments?
> 
> 
> _______________________________________________
> vtk-developers mailing list
> vtk-developers at vtk.org
> http://www.vtk.org/mailman/listinfo/vtk-developers