[Ctk-developers] Scenes

[Stephen Aylward]

Hi Steve,

Very interesting ideas!

Really good point: why bother creating a scene that depends on Qt if
an ITK/VTK implementation is possible.   I used  "ITK/VTK" as the
placeholder since I'm still working out the pros and cons of using
each - I just read your email and I need to think about it more.

One question that quickly jumps out at me...given that the algorithms
are in ITK, perhaps it would be better to have the scene in ITK?  That
way algorithm developers don't need any other libraries.   If data and
logic are in ITK, then the chance of others following our lead might
be greater since the cost to learn/use our methods/algorithms is then
minimized.    We can use ITK's IO library which supports more medical
image types than VTK's IO library.  Also consider that although CTK is
committed to VTK, perhaps if we can keep our methods and data
independent of it, then the XIP/OpenInventor folks might benefit more
directly and with less refactoring.

Stephen





On Sun, Sep 20, 2009 at 11:54 AM, Steve Pieper <pieper at bwh.harvard.edu> wrote:
> Hi Guys -
>
> Sorry to be late to the conversation, but I did want to add a few points
> from our experience when redesigning MRML a few years ago.
>
>
> * Object model and data structures for the scene:
>
> A core issue for MRML was the on-disk and in-memory representation of
> complex data types that are part of the scene.  Since we use both VTK and
> ITK extensively, we considered creating a toolkit-independent object layer
> with adaptors to both toolkits... but we quickly realized that this meant
> building not only a new object layer but two complete adaptors.  Instead, we
> standardized on VTK for our MRML Scene layer and have an interface to ITK (a
> pretty slick one - it uses the ITK I/O factory to give pointer access to the
> image data).
>
> By sticking, in particular, with vtkPolyData we have been able to represent
> things like diffusion tractography fiber bundles with tensors-per-vertex and
> annotated brain surface models.  These are hard data structures to write and
> debug, and also by using VTK as the object model it plugs easily into the
> rest of the application.  Note that MRML *does not* describe the GUI and it
> *does not* even link to OpenGL.  It merely uses VTK for the data structures
> and the rich set of I/O and data manipulation utilities.  Since CTK will
> have VTK available, the same considerations apply.
>
> One way of interpreting this approach is that it is essentially Pat's
> suggestion (Implementation 4), but using VTK (mostly vtkCommon) as the basis
> for the internal data structures.  From there you can have an ITK adaptor
> and/or a Qt signals/slots interface as Stephen has shown in mavScene.  Using
> QtCore's tree data structures and other tools to organize the vtkObject
> instances may be an ideal combination.
>
>
> * Descriptions of Image Acqusition
>
> We should distinguish between "objects that fill space" and "physical
> observations of objects using particular imaging devices and protocols" --
> we typically have the latter and are trying to calculate the former.  The
> ctkScene described on the wiki doesn't (yet) talk about the importance of
> having the scene be able to describe the image acquisition, clinical
> information, or other real-time info.  Image acquisition info is basically
> what comes from the DICOM header like the imaging device and protocol.  In
> general we should also be able to represent data on things like fMRI
> stimulus presentation and physiological response if ctkScene apsires to have
> complete information necessary for a wide range of image analysis tasks.
>
>
> * The "physical scene" vs. the "semantic scene"
>
> The current ctkScene description focuses on objects in space, but there is
> also a lot of non-physical information that we typically embed in the scene
> - things like anatomical labels and annotations.  In MRML we allow /model
> hierarchies/ independent of /transform hierarchies/ (e.g. the "skeletal
> system" as opposed to the "hand").  The former might be used for
> visualization, while the latter can be used for positioning child objects
> relative to parent objects.
>
>
> In general I agree with the sentiment that these are complex issues that are
> best discussed over a beer (or two...).  See you in Oxford!
>
> -Steve
>
> p.s. I added these to the wiki even if they are too late to be included in
> Stephen's summary.
>
> Stephen Aylward wrote:
>>
>> Hi,
>>
>> I am sorry - I am doing a very poor job of explaining IPScene versus
>> CGScene.    I've caused you to confuse IPScene with ITK and VTK
>> processing pipelines.
>>
>> The IPScene represents objects in physical space, not processing
>> pipelines.  Each node in the IPScene represents an object (e.g., a
>> chair) that has a location and occupies a volume in some physical
>> coordinate system (such as an ultrasound probe in an operating room or
>> the liver in the abdomen of an individual).   You can ask if a
>> physical point is inside of an object, you can move an object (node)
>> in physical space by applying a transform to the matrix associated
>> with that node (i.e., the matrix the positions the node with respect
>> to its parent note).    A palm of a hand may the root of one IPScene
>> with the fingers as branches off of the root.  If you move the palm by
>> changing its transforms, the fingers (as children) move with it in
>> space.
>>
>> Below you seem to be describing IPScenes as processing pipelines
>> between modules/vilters.  That is something quite different - both VTK
>> and ITK call them pipelines.   Such pipelines implement algorithms.
>> If we want to represent pipelines generically, I fully agree that a
>> much different conversation would be needed.
>>
>> My intent was to distinguish GUI elements from physical objects.   For
>> example, in a CGScene, you would represent a GUI element, such as the
>> visual representation of a pipeline.  While an IPScene is the data you
>> would pass between filters/modules in a pipeline.
>>
>> Sorry - not certain if I am helping.   I look forward to our
>> face-to-face discussions.
>>
>> Stephen
>>
>>
>> On Fri, Sep 18, 2009 at 8:00 PM, Paladini, Gianluca (SCR US)
>> <gianluca.paladini at siemens.com> wrote:
>>>
>>> As I mentioned, Open Inventor supports both types, CGScenes and IPScenes
>>> - it is not limited to just the first kind.
>>> This is why it was relatively simple to wrap ITK and VTK in Open
>>> Inventor, even though those libraries have nothing to do with CGScenes.
>>>
>>> In XIP Builder CGScenes are built top-down using connections in the
>>> top/bottom part of the box representing a node, while IPScenes are usually
>>> arranged freely on the visual workspace and their connections emanate
>>> laterally from the left and right edges of the box. IPScene connections are
>>> type-specific and are created by popping up a "field panel" for the object
>>> and connecting input-output parameters. Open Inventor supports user-defined
>>> field converters so that some connections between fields of different type
>>> (eg. text vs. numeric) are converted automatically.
>>>
>>> When you start digging in, there is a mountain of details that need to be
>>> supported and defining a toolkit-agnostic scene graph may be impossible to
>>> attain. That's why I'm an advocate of not re-inventing what others have
>>> spent decades refining, and I prefer just creating an XML file definition
>>> for toolkits that exist.
>>>
>>> I am beginning to wonder if we are better off defining a common XML file
>>> format that contains toolkit-specific declarations. This would save us a lot
>>> of time, because we would not have to figure out how to create generic scene
>>> descriptions. The file could declare that, in order to parse a particular
>>> scene you will need to have, for example, VTK, MITK and IGSTK. The
>>> application would make sure those DLLs are loaded, then parse the file and
>>> instantiate those objects. We already have to do this in XIP Builder because
>>> various objects are organized in separate libraries, and those libraries
>>> need to be present at runtime otherwise certain scene graphs cannot be fully
>>> loaded (objects from missing libraries are shown as red boxes).
>>>
>>> If we go with this approach then the focus in CTK would shift towards
>>> implementing "bridge" scene objects that allow connections between different
>>> toolkits - objects that would accept an input from an XIP object and provide
>>> a VTK output, and so on. As long as CTK provides a library of such objects
>>> and a file format to define how to create scenes that intermix the use of
>>> all out toolkits, our goal for interoperability would be met.
>>>
>>> -----Original Message-----
>>> From: Stephen Aylward [mailto:stephen.aylward at kitware.com]
>>> Sent: Friday, September 18, 2009 7:01 PM
>>> To: Paladini, Gianluca (SCR US)
>>> Cc: Wolf Ivo; ctk-developers at commontk.org
>>> Subject: Re: [Ctk-developers] Scenes
>>>
>>> Hi,
>>>
>>> Gianluca and Ivo bring up a good point..."what is a scene?".   The
>>> wiki begins with that discussion and a list of requirements, and it seems
>>> as though we might need to hash it out a bit more.
>>>
>>> To being, perhaps we need to distinguish computer graphics scene graphs
>>> ("CGScenes") from image processing scene graphs ("IPScenes").
>>> The two serve different purposes.
>>>
>>> CGScenes are used in two of the largest, community imaging
>>> environments: XIP (OpenInventor) and Slicer (MRML).   CGScenes use
>>> traversal-based semantics to define the context at a node, this is
>>> motivated by openGL's use of contexts during rendering.   CGScenes are
>>> highly developed and have wonderful utility in computer graphics, and
>>> clearly they are a solid basis for imaging applications.
>>>
>>> IPScenes have also been developed independently by several groups (MITK,
>>> IGSTK, itkSpatialObjects, and OpenMAF).  An IPScene is a tree of transforms,
>>> where the transform at a node only influences its children.  That is, you
>>> don't traverse the scene graph to understand
>>> the context of the image stored at a node.   IGScenes have
>>> demonstrated themselves to be useful for representing the relative
>>> location of patients and instruments for surgical guidance systems, the
>>> interactions of physiological systems for gate/motion analysis and
>>> prosthesis design, the multiscale representation of biological processes for
>>> multi-system disease modeling, and numerous other
>>> application areas.   Sorry if my bias is showing here.
>>>
>>> As Gianluca mentioned, we need to save the state of an application as
>>> it processes its data.   CGScenes make this easy.  The centralization
>>> of GUI and model information simplifies saving state and simplifies
>>> the implementation of undo/redo capabilities.   However, it could be
>>> argued that such mingling does confound the exchange of data across
>>> different applications and does create massive scenes that mix physical and
>>> GUI constructs.  Defining a common API for such disparate types of data does
>>> perhaps necessitate the over simplification of the
>>> API.   For example, the scene requirements listed on our wiki suggest
>>> that we need to be able to ask for the physical location of an
>>> object...how can you ask a GUI button for its physical location?   How
>>> can you apply a 3D transform to a slider?   The notion of saving the
>>> state of an application is critical, but perhaps it is better suited for
>>> a data structure other than the scene graph; one that has its own, specific
>>> API.
>>>
>>> So, we need both: a scene of objects and a record of the GUI state.
>>> On some implementations, it may be useful to combine them as
>>> facilitated by CGScenes.   However, we should also allow other
>>> implementations to keep them separate.
>>>
>>> In particular, perhaps the XIP can be used as a reference
>>> implementation demonstrating their combined storage.    Additionally,
>>> it would be interesting to see if MITK, IGSTK, or others can be adopted
>>> as a companion implementation that demonstrates the utility of keeping GUI
>>> and physical objects in separate data structures.
>>>
>>> Let's begin the talk on Friday with "What is a scene"   - I fear we
>>> won't get beyond this question unless we have already had a few beers,
>>> and it will be a bit early in the day.   So perhaps we should try to
>>> limit the time for each topic so that we can hit some potentially fertile
>>> topics for discussion as well as the more cantankerous topics.
>>>
>>> Thanks,
>>> Stephen
>>>
>>> On Fri, Sep 18, 2009 at 5:56 PM, Paladini, Gianluca (SCR US)
>>> <gianluca.paladini at siemens.com> wrote:
>>>>
>>>> The advantage of a scene graph comes primarily from those nodes which
>>>> define attributes rather than drawn objects, because those attributes
>>>> can be inherited in lower branches of the scene and represent the
>>>> state of the system at any point during scene graph traversal - that
>>>> is a key feature, especially when it comes to transform matrices.
>>>> "Separator"
>>>> objects prevent those state changes from propagating, while "Group"
>>>> objects don't. It is this mechanism that makes scene graphs ideal for
>>>> setting up coordinate systems and controlling parts that move with
>>>> respect to each other.
>>>> Without that it's not a scene graph, it's a pipeline. Pipelines are
>>>> supported in Open Inventor too, but via objects called engines rather
>>>> than nodes.
>>>>
>>>> -----Original Message-----
>>>> From: ctk-developers-bounces at commontk.org
>>>> [mailto:ctk-developers-bounces at commontk.org] On Behalf Of Wolf Ivo
>>>> Sent: Friday, September 18, 2009 5:02 PM
>>>> To: Stephen Aylward; ctk-developers at commontk.org
>>>> Subject: Re: [Ctk-developers] Scenes
>>>>
>>>> Hi Stephen,
>>>> As for the MITK team, we traditionally like the type of definition of
>>>> a scene that you gave and the mode-view-controller pattern. Both is
>>>> pretty close to what is used within MITK (except for some extensions,
>>>> which would be great to have) and therefore could nicely be integrated
>>>> from CTK. Nevertheless, my feeling is that starting with a scene
>>>> implementation is not the easiest way to go.
>>>>
>>>> And caution (correct me, if I am wrong): The Open Inventor definition
>>>> of a scene is different from yours (and ours)! Nodes in an Open
>>>> Inventor scene graph are not necessarily 'objects' (called shape nodes
>>>> there), but can also be lights, materials (e.g. color), transforms
>>>> (matrices), switches etc. Thus, the definition "a scene is composed of
>>>> objects that fill a portion of space and time" is not compatible with
>>>> Open Inventor (a material node does not fill a portion of space).
>>>> Maybe it is possible (and desirable) to combine both ideas of defining
>>>> a scene graph: the more data-centered approach defined above and the
>>>> visualization-centered approach of Open Inventor. But we should be
>>>> careful that we are talking about the same things (a topic for the list
>>>> of questions).
>>>>
>>>> Regarding a brief presentation, we recently added some semantic
>>>> information to our scene description, which might be useful for ctk,
>>>> too.
>>>>
>>>> Best regards,
>>>>
>>>> Ivo
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> Von: ctk-developers-bounces at commontk.org im Auftrag von Stephen
>>>> Aylward
>>>> Gesendet: Do 17.09.2009 02:18
>>>> An: ctk-developers at commontk.org
>>>> Betreff: [Ctk-developers] Scenes
>>>>
>>>>
>>>>
>>>> Hi,
>>>>
>>>> ctkScene:  Love it?  Hate it?  Know you can make it better?
>>>>
>>>> Regardless of your current attitude, please send me your ideas and
>>>> presentations!!!
>>>>
>>>> I'd like to help organize the discussion on ctkScene that is scheduled
>>>> for next Friday.
>>>>
>>>> 1) If you are interested in making a brief presentation (less than 10
>>>> minutes), then please send me an email with the title.   I will add it
>>>> into our agenda.
>>>>
>>>> 2) If you have revisions to the Wiki page on ctkScene - please make
>>>> those revisions by 5pm EST (22:00 GMT) on Friday.   I am going to
>>>> summarize those wiki pages as a set of slides at the start of the
>>>> meeting.
>>>>     http://my-trac.assembla.com/protoctk/wiki/ctkScene
>>>>
>>>> 3) If you have questions, concerns, needs, wants, or rants regarding
>>>> the ctkScene, please send them to me and I will create a set of slides
>>>> that list "open-issues" to help focus our discussions.   We only have
>>>> one-hour, so I'd like to focus on high-level issues that remain, prior
>>>> to getting into implementation details.   For example, should a
>>>> ctkScene be only a file format or should it also be implemented as a
>>>> set of C++ classes?   Should a ctkScene be specific to or independent
>>>> of the visualization library?   Should a ctkScene be our primary data
>>>> structure, or should our primary data structure simply be an image?
>>>> ...please add to this list of questions...
>>>>
>>>> Thanks!
>>>> Stephen
>>>>
>>>> --
>>>> Stephen R. Aylward, Ph.D.
>>>> Director of Medical Imaging
>>>> Kitware, Inc. - North Carolina Office
>>>> http://www.kitware.com <http://www.kitware.com/> stephen.aylward
>>>> (Skype)
>>>> (919) 969-6990 x300
>>>> _______________________________________________
>>>> Ctk-developers mailing list
>>>> Ctk-developers at commontk.org
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/ctk-developers
>>>>
>>>>
>>>> _______________________________________________
>>>> Ctk-developers mailing list
>>>> Ctk-developers at commontk.org
>>>> http://public.kitware.com/cgi-bin/mailman/listinfo/ctk-developers
>>>>
>>>
>>>
>>> --
>>> Stephen R. Aylward, Ph.D.
>>> Director of Medical Imaging
>>> Kitware, Inc. - North Carolina Office
>>> http://www.kitware.com
>>> stephen.aylward (Skype)
>>> (919) 969-6990 x300
>>>
>>
>>
>>
>



-- 
Stephen R. Aylward, Ph.D.
Director of Medical Imaging
Kitware, Inc. - North Carolina Office
http://www.kitware.com
stephen.aylward (Skype)
(919) 969-6990 x300