SceneGraphExamples

From IGSTK

Contents 1 Illustration (IntraOp scenario) 2 A simple example 3 A more complex example 3.1 PreOp 3.2 IntraOp 3.3 PostOp 4 Intended use 4.1 ImageReslice(SORep): 4.2 VolumeRender(SORep): 4.3 AnatomicalSurfaceModel(SORep): 4.4 TrackerToolSurfaceModel(SORep): 5 Previous comments: 5.1 Patrick: 5.2 Frank: 5.3 David: 5.4 Minutes 082307:

Illustration (IntraOp scenario)

Abbreviations (Remember to update the names (coordinate-systems and transforms) when a common agreement have been achieve)

• Coordinate systems (CS):
  • gl = global CS = tracking system CS (optical: camera CS, magnetic: transmitter CS)
  • tf = tracking-frame CS
  • rf = reference-frame CS (tf for the reference attached to the patient)
  • tt = tool-tip CS
  • v = volume CS
  • us/mr/ct = v CS
  • p = patient CS
• Transforms (T):
  • TT = Tracking T
  • CT = Calibration T
  • VT = Volume T
  • PasRT = Patient Registration T
  • RCT = ReConstruction T
  • V2VRT = Volume2Volume Registration T
• Others:
  • P = Point

A simple example

Pseudo code for reading DICOM data and registration to reference frame / tracking system / patient:

• ImageSpatialObject CTImage
• CTImage.RequestReadDICOM( directoryName )
  • Create VT from DICOM header data (-> dfc_VT_vfc-CTImage)
  • CTImage.RequestSetVT( dfc_VT_vfc-CTImage ) Note: No parent, implicitly this could means that the transform is relative to the "referance" for this phase (dfc here).
• Perform Image2Patient (or Image(vfc)2Trackingsystem(rfc or gfc)) registration (->rfc_VT_vfc-CTImage)
  • CTImage.RequestSetVT( rfc_VT_vfc-CTImage ) Note: The VT for CTImage is changed, no parent meaning that the transform is relative to the reference of the intra.op. phase (rfc) and not relative to another Spatial object.

A more complex example

PreOp

Reading DICOM data:

• Pseudo code:
  • ImageSpatialObject MRImage1, MRImage2, CTImage
  • MRImage1.RequestReadDICOM( directoryName )
    • CreateVT and SetVT (MRImage1.RequestSetVT( dfc_VT_vfc-MRImage1 )
• Figure AfterReadingPreOpData

Registration of all PreOpData to a master volume (MRImage1):

• Pseudo code:
  • Perform MRImage2 to MRImage1 V2V reg. (-> vfc1_VT_vfc2 )
  • MRImage2.RequestSetVT( vfc1_VT_vfc2, MRImage1 ) Note: Parent here, so the transform is set relative to the this.
  • It should be possible to serialize the data at this point (the work presented here is done in the office the day before surgery, the data is serialized, the nav.sys is shot down and in the OR the next morning the data is de-serialized and read into the system (i.e. it is not necessary to build the sene-graph once more)
• Figure AfterV2MasterVReg

IntraOp

Registration of PreOpData to patient:

• Pseudo code:
  • Perform Image (vfc-MRImage1) to Referanceframe/Trackingsys/Patient(rfc) reg. (-> rfc_VT_vfc)
  • MRImage1.RequestSetVT( rfc_VT_vfc )
• Figure AfterPasReg

Intra-operative data acquisition (USImage/MRImageIntraOp/CTImageIntraOp):

• Pseudo code:
  • Perform ultrasound acq. and 3D us-reconstruction to produce USImage and rfc_VT_vfc-USImage.
  • USImage.RequestSetVT( rfc_VT_vfc-USImage )
• Figure AfterUSAcq

Registration of PreOpData to IntraOpData:

• Pseudo code:
  • Perform MRImage1 to USImage reg. (-> us_RT_mr )
  • MRImage1.RequestSetVT( NEW(rfc_VT_vfc-MRImage1 ) Note: the VT is updated based on the OLD( rfc_VT_vfc-MRImage1 ) and the found us_RT_mr)
• Alternative:
  • MRImage1.RequestSetVT( us_RT_mr, USImage ), see SceneGraphExamplesAlternatives
• Figure AfterMR2USReg

PostOp

Registration of PreAndIntraOpData to PostOpData:

• Pseudo code:
  • ImageSpatialObject MRImagePostOp
  • MRImagePostOp.RequestReadDICOM( directoryName )
  • Perform MRImage1 to MRImagePostOp reg. (-> vfcpost_RT_vfc1)
  • MRImage1.RequestSetVT( vfcpost_RT_vfc1, MRImagePostOp )
  • USImage.RequestSetVT( vfc-mr_VT_vtc-us, MRImagePostOp ) Note: based on found RT and old VT
• AfterV2VReg

Intended use

• In the figures above the SceneGraph is used to organize pre- intra- and post- op. data from modalities like Ultrasound, MR and CT, i.e non-visible SpatialObjects of the following types: Image(SpatialObjects) (as well as SegmentedBinaryImage(SO) and ROI-Image(SO)) and AnatomicalSurfaceModel(SO).
• Representations (e.g. ImageResliceRep, visual scene graph, vtk) and Tracker/TrackerTool obj. are not included in order to keep things simple (they use functionality provided by the SceneGraph instead)
• The different TrackerTools should be given graphical representations and rendered together with image data and AnatomicalSurfaceModels, this requires a TrackerToolSpatialObject and a TrackerToolRep. In order to make the SceneGraph consistent (spatial object org.), also the TrackerToolSpatialObject should be placed here. One option would be to make the TrackerTool a Spatial object (Tracker->TrackerToolSO (combine TrackerTool->TrackerToolSpatialObject)->TrackerToolRep->View) that can be visualized using different representations (e.g. 2D: cross, 3D : TrackerToolSurfaceModelSOR)
• In my head, basically four types of "representations" exists (the knowledge that a given object is a SpatialObjectRepresentation (SOR), or a SpatialObject(SO) for that sake, could be made more implicit in order to keep the names shorter)

ImageReslice(SORep):

• ImageResliceRep irr
• irr.RequestSetImageSO( imageSO )
• irr.RequestSetTrackerTool(SO??)( trackerTool )
• It should be possible to:
  • irr.RequestSetImageSO( imageSO_2 ), i.e. use another image, e.g. point at the patient (given location) and go thought all available data.
  • irr.RequestSetTrackerTool(SO??)( trackerTool_2 ), i.e. use another tool, e.g. the surgeon change from on tracked instrument to another.
• Algorithm for extracting a slice from an image based on the current location of the tip of a tracked tool
  • Get the latest tool tip position in reference frame coordinates (rfc) from trackerTool, i.e TT_rfc
  • Use the latest rfc_VT_vfc from imageIO, use this to convert TT_rfc into TT_vfc (TT_vfc = inv(rfc_VT_vfc)*TT_rfc) and use this to extract the slice at the desired location.
    • imageSO use the scene graph in order to calculate the the total rfc_VT_vfc (depending on the location of the image in the tree, this could be a concatenation of 0 to many transforms).
    • In the name of efficiency it might be wise to have the transform that takes you from the volume all the way down to the reference stored in a separate variable so that a re-concatenation in not needed each time an update is issued from the view (the total transform must be updated for all the SO in the subtree below the change.
    • In addition it would be very nice if we could have a list of all these transforms making it possible to go back in time or toggle a pre2intra op. reg. transform on and off.

VolumeRender(SORep):

• Is direct volume rendering of image data contained in a scene graph ok???, please comment Luis.
• In the future, we would like a GPU based volume renderer capable of rendering multiple overlapping volumes where transfer-functions and clipping-planes can be set individually for each volume.
  • VolumeRender.RequestAddImage( ImageSO_1 )
  • VolumeRender.RequestAddImage( ImageSO_2 )

AnatomicalSurfaceModel(SORep):

• Should be ok

TrackerToolSurfaceModel(SORep):

• See comments above

Previous comments:

Patrick:

This is an excellent graphical illustration of the scene graph concept in surgery.
As we discussed in the last tcon, we will:

1. Use Tracker Frame, Reference Frame, and Data Frame (or Volume Frame)
2. Take Tracker out of the scene graph tree
3. Assume we have a single tracker in application right now, may develop composite Trackers class in the future.
4. The PatientToImageTransform in old API should now be associated with a ImageSpatialObject instead of Tracker, to be able to accommodate multiple image volumes in the application
5. Have a single reference in the appliaction
6. Remove coordinate system from SpatialObject and Tracker class

Please add anything I have missed here.

Luis, could you work with Frank to carry out the refactoring? I am not sure if this is clear enough for you to continue or not. I think Frank could take the lead in naming and defining the API. Let's move this forward and try to close the release in this month.

Frank:

Some comments before the Tcon: I will try not to repeat the things that have already been said:
http://public.kitware.com/IGSTKWIKI/index.php/Surgical_coordinate_changes
http://public.kitware.com/IGSTKWIKI/index.php/Talk:Surgical_coordinate_changes

• Coordinate systems, transformations and spatial objects (as well as tracker and trackerTool) should be treated as separate concept whenever possible.
• Names known from IGS should be used whenever possible
• To keep it simple (and familiar) (at least for the user) should be a very important design criteria.
• The design should be based on how it's supposed to be used in most cases (95%, default behavior, with enough flexibility to deal with the other cases, not more)
• It would be nice to test the proposed framework on a concrete and common scenario (figures and pseudo-code)
• What is regarded as World / Global reference coordinate-system change from phase to phase in the surgical workflow:
  • PreOpRefSys: e.g. the "DICOM" coord. sys. of the "master volume"
  • IntraOpRefSys: the (Patient)Ref.Frame (a special TrackerTool, most common, default) or the coord. sys. of the Tracker itself (e.g. middle of the camera for optical systems)
  • (PostOpRefSys: e.g. the "DICOM" coord. sys. of one of the post. op. volumes)
• The sub-tree/hierarchy of image date basically moves through these phases (e.i. the world coord. sys. change but the sub-tree remains somewhat similar.
• The change from one phase to the next happens as a result of some kind of registration. e.g. Patient registration (from PreOpRefSys to IntraOpRefSys)
• Changes (in position) within the image sub tree happens as a result of brain-shift calibration for example.
• Well known names for well known procedures should be used: SetCalibration, SetPatientRegistration, etc.

David:

• A scene graph can be used, but there in the discussion on the wiki page it is not clear at all how the different elements of the graph apply to image-guided surgery.

• A good example of that is the "WorldFrameObject." After reading the page, I really have no idea what the coordinate system of the WorldFrameObject is. Is it in the coordinate frame of the tracker reference?

• In a way the whole wiki page is backwards. It has to begin with the concepts that are required for image-guided surgery, and then explain those concepts concretely in terms of a scene graph. Instead the scene-graph idea is presented and it is left up to Frank to figure out exactly how it is going to be applied to surgical concepts.

Minutes 082307:

Spatial Object refactoring in conjunction with Tracker interface changes and Surgical coordinate changes

• Luis's report on progress:
  • Misleading API of the Transform::TransformCompose() method. A = Transform::TransformCompose( B, C ) versus A.TransformCompose( B, C)
  • CoordinateReferenceSystem class added
  • WorlCoordinateReferenceSystem class added
  • All SpatialObjects must now be attached to a parent
  • Only the WorlCoordinateReferenceSystem can be without parent
  • All SpatialObjects must provide a transform relative to the parent
  • SpatialObject can be located only once (Transforms are Final)
  • Tracker now has an internal CoordinateReferenceSystem instance
  • TrackerTool now has an internal CoordinateReferenceSystem instance
  • TrackerTool reference system is a child of the Tracker reference system.
  • Tracked Spatial Object is a child of the TrackerTool reference system
  • TrackerTool: Calibration, Transform get build by the Composition of child, parent
• Discussion:
  1. Created a scene graph for coordinate system
  2. When introducing a new object, you need to attach it to another reference system (a parent)
  3. Converted some tests in the Sandbox, added CMake option to use the new coordinate system to keep main repository testing stable
  4. Tracker has its own internal coordinate system object which is not exposed
  5. View is going to request SpatialObject for latest position, ComputeTransform. Then it will traverse the tree to update transforms.
  6. Some terminologies used in the tracker will have to change
     • The CalibrationTransform will the transform set to the SpatialObject when it is attached to a TrackerTool
     • There will be no more PatientImageTransform, where to set the registration results will depends on the reference system
  7. Patrick's comments:
     • This will seem to be a little counter-intuitive to people who has worked in this field for a long time
     • It will be nice to have graph representation on how to perform the registration under the new framework, and how to set it in the SpatialObject hierarchical tree
     • It will be nice to have different examples to demonstrate the usage of this new framework for registration under all possible scenarios
     • When Attach to Parent, make sure it only has one single parent
     • Should have a safety check to detect loop
     • So far we don't see the need for detaching and re-attaching the spatial object to another parent. Will not implement this for right now
  8. Frank's comments:
     • It seemed a little bit more complex than conventional way, but it is very flexible.
     • Suggest to provide more user friendly API methods
     • Why not make reference system object to be inside SpatialObject?
       1. Remain like this for right now
       2. Will take a closer look into the pros and cons
  9. Luis's question
     • Should the transform for SpatialObject be set only once?
       1. Static object should only be placed in the scene once and then fixed
       2. Frank: Should have the option to move spatial object around