DesignChallenges

From IGSTK

Jump to: navigation, search
  • An IGS Application of the future will be complex, involving pre.op., intra.op. and real-time data, multi-modal slicing and volume rendering visualization techniques as well as the possibility to update pre.op. data using intra.op. data.
  • Many challenges exists in designing such a complete system.
  • Things like current and future requirements, all acquired experience in the field of IGS, etc. must be taken into consideration at the same time in order to come up with a toolkit that is safe, efficient (both in terms of CPU(GPU) and RAM(VRAM) usage) and last but not least provides easy accessible, high level, advanced functionality.
  • This is an early attempt to address some of these challenges and to start a discussion as to how these challenges are best solved.


Contents

Coordinate systems / Transforms / Registration

  • Challenge: To handle all these cases:
    • Only one pre.op. dataset (CT/MR)
    • Many pre.op. datasets.
    • Only intra.op. 3D data (US)
    • Many pre.op. (CT/MR) and many intra.op. (US) datasets.
    • Rigid V2V registration to "update" pre.op. data using intra.op. data
    • Non-rigid (deformable) V2V reg.
    • etc.
    • When the necessary transforms are found between all preoperative data it should be possible to put that information back into a system that "takes care of further processing".
    • When the transform that updates preoperative data using intra-operative data is found it should be possible to put that information back into a system that "takes care of further processing"
    • etc.
  • Possible solution:
    • Every volume is associated with a volume-transform (VT).
    • For the pre.op. dataset registered to the patient the VT is equal to the patient registration transform, and the volume coord. sys. (v) is equal to the patient coord. sys. (p).
    • When anatomical shift compensation is performed on a pre.op. dataset only it's VT is updated.

Fig. 1: Top: Coordinate-systems & transforms, Bottom: Image-Registration and Transforms

  • Abbreviations:
    • Coordinate systems (CS):
      • gl = global CS = tracking system CS (optical: camera CS, magnetic: transmitter CS)
      • tf = tracking-frame CS
      • rf = reference-frame CS (tf for the reference attached to the patient)
      • tt = tool-tip CS
      • v = volume CS
      • us/mr/ct = v CS
      • p = patient CS
    • Transforms (T):
      • TT = Tracking T
      • CT = Calibration T
      • VT = Volume T
      • PasRT = Patient Registration T
      • RCT = ReConstruction T
      • V2VRT = Volume2Volume Registration T
    • Others:
      • P = Point


Update efficiency / SurgicalSpace-ImageSpace delay / "Temporal safety"

  • Challenge: To have a responsive system (i.e. the display presented to the surgeon is an accurate picture of the surical situation, not only spatially but also temporally) when new complex features are added:
    • Video grabbing and display (optimal synchronization, no conversion if possible and grab to texture are things that might help on the comp. level)
    • 3D freehand reconstruction (optionally while grabbing)
    • VolumeRendering (of multiple overlapping volumes)
    • V2V Registration for anatomical shift compensation (optinally while navigating)
    • etc.
  • Possible solution:
    • Look at data/push/event vs. view/pull driven update, e.g. letting the Tracker tell all the views when to update instead of the PulseGenerator for each view
    • If new data are available they should be displayed ASAP, if not the computer should rest (available for background prcessing)

Fig. 2: View/Poll vs. Data/Push driven update

  • Abbreviations:
    • V = View (V_A = View A)
    • SO/SOR = Spatial Object / Spatial Object Representation (SO_1 = SO 1, SOR_1B = SOR 1B)
    • T = Tracker
    • VG = VideoGrabber
    • ----
    • PG = Pulse Generator
    • PG_VA = PG for View A
    • PG_T = PG for Tracker
    • PG_VG = PG for VideoGrabber
    • T_HW = Tracker HW (i.e. system/device)
    • T_B = Tracker Buffer (T_HW / T_B communication in dedicated thread)
    • T_T = Tracker Tool
    • VG_HW = VideoGrabber HW (i.e. FrameGrabber (PCI, FireWire, USB2))
    • VG_B = VideoGrabber Buffer (VG_HW / VG_B streaming in dedicated thread)


Multimodal Visualization

  • Challenge: In MultiModViz we need access to more than one medical image volume (image source) at the time (can not be cotained in the SOR/SO), e.g.
    • A (Complex) SlicerSO might need access to a pre.op. volume and RT-2D ultrasound stream or a pre.op. and a intra.op. volume (see fig.)
    • A (Complex) VolumeRenderSO that can render multiple overlapping volumes need access to these volumes.
  • Possible solution:
    • Seperate the volume data from the spatial objects (SO/SOR).

Fig. 3: Multi-modal visualization

  • Abbreviations (se fig 2):
    • V = 3D volume data (CT/MR/US)
  • Comments:
    • Spatial Objects (SO) = Spatial Temporal Objects (STO) = Spatial Temporal Visual Objects (STVO) = Visual Objects (VO)
    • Basically three kinds of Visual Objects (VO) can be identified in an IGS application.
      • SliceVO (slicing through a 3D volume)
      • SurfaceVS (GeometricVO=TriangleVO), can further be subdivided into:
        • Anatomical/Pathological structure (segmented from a 3D volume, we need a ref. to this volume even thought the segmentation is done outside the navigation application)
        • Virtual Tool Object (Virtual representations of the physical surgical tools used (e.g. pointer, us-probe, etc.), information about the calibration transform, etc. are needed in order to make the virtual object realistic).
      • Volume (rendered) Visual Object: Volume rendering of multiple (potentially overlapping) volumes (potentially different spacing and size) that are potentially translated and rotated relative to each other.
      • In addition, there will be a need to stream data to a plane when the VideoGrabber comp. is ready,this is visually similar to slicing.
    • Some of these Visual Objects (SliceVO/VolumeVO) need access to multiple volumes at the same time in order to perform multi-modal viz.


Volume Rendering functionality

  • There is a need for a VolumeRendering comp. (of multiple overlapping volumes), especially for vascular procedures.


Slicing

General comments:

  • This is a very good initiative and based on limited practical experience with existing functionality for slicing in IGSTK (igstkImageSpatialObjectRepresentation and igstkObliqueImageSpatialObjectRepresentation) the proposed class is definitely a step forward in terms of functionality that is required clinically (in general, not just for the needle biopsy app. etc.)
  • Slicing have been and will probably remain the main type of "navigation map" in any IGS app. Tracked surgical tools define how the available volumes should be sliced ("automatic/tracked mode") (part from the initial planning phase, i.e. before any volume have been registered to the patient ("manually/mouse mode")).
  • In that regard one can say that the slicing component is the most important class in any IGS app./toolkit, i.e. this is core functionality, other components should support this in order to make slicing shine. This should not be treated as a special case.
  • When designing a new component (or toolkit) it's important to get ALL clinical requirements on the table first. Only when core requirements are processed and agreed upon should we start to look at the optimal way to implement this functionally in software, taking things like simplicity (as seen from the application developers' point of view (increased use of IGSTK...)) and safety into account. Together I think that we have the necessary clinical know-how (close collaboration with surgeons, some of us have many years of OR experience, etc.) and in terms of slicing-design, lets look at how this has been done before, e.g. Davids' atamai-python-classes, etc.
  • In IGS "continuous" slicing is needed (so we should focus on this), "index"-based slicing might be beneficial in some special cases during manual mouse controlled mode before surgery. However, as interaction in the window is a more intuitive way then sliders to pick the desired slice, and the window pixel index rarely coincides with the image indexes, it's not unnatural to also think of this process as "continues". In addition there is the benefit of interpolated slicing when zooming (in order to localize a landmark in an image) or generating slices that are not scanner-aligned.
  • I.e. the same slicer object could be used in both pre.op. manual-mouse-controlled mode and intra.op. automatic tracker-controlled mode. This is especially beneficial as there is a need to switch between manual and automatic mode during surgery to (see below). If this is not possible do to IGSTK-design-conventions, the mouse-based slicer object and the tracker-based slice object should have the same superclass do to the fact that apart from the way the slices are extracted (mouse or tracker) the rest would probably be the same.
  • If a special tracker-based view class really is necessary, this class should become the core view class (after all, tracking is core functionality in an IGS app....), and other view classes should be treated as special.




Fig.1: Volume-based orthogonal slicing (top). Tool-based orthogonal slicing (bottom). Called "Anyplane" in Trondheim. A plane perpendicular to the tool is not shown (triple anyplane).



Fig.2: Generating and displaying slice data. Orthogonal (A) or oblique (D) slices relative to the volume axis can be controlled by a surgical tool (intraoperatively) or by a mouse (pre- and postoperatively). The extracted slices can be displayed directly in a window (B, E), or texture-mapped on polygons in a 3D scene and rendered into a window (C, F)

Slicing: Types/Relative to (i.e. orthogonal-slicing relative to the volume, patient, surgical tool and surgeon)

  • Volume-based-slicing (VBS)


Fig. 3: Traditional mouse-controlled volume-based slicing: Point/drag in the axial-view and the coronal and sagital views are updated accordingly (no sliders). More than a single pre.op. volume available, corresponding views: Orthogonal slicing relative to the master volume often mean oblique slicing relative to the slave volume. Need for synchronization, for example between axial, coroanl and sagital views for volume one (row 1) and for corresponding axial views from volume 1 and volume 2 (column 1).

  • Patient-based-slicing (PBS) (VBS is usually the same as PBS for pre.op. data, but not necessarily for intra.op. ultrasound)


  • Tool-based-slicing (TBS) (our surgeons uses TBS more then VBS/PBS, but they change all the time during a procedure.


Fig. 4: Tracker controlled tool-based slicing (see fig. 1, dual anyplane).

  • Surgeon-based-slicing (SBS) (requires some kind of tracking of the surgeon, so a bit difficult today, but maybe the right way to go in the future)


  • Comments:
    • The orthogonal vs. oblique slicing (relative to a single volume) discussion seems a bit strange, especially in a multi modal setting. Orthogonal slicing relative to a surgical tool usually means oblique slicing relative to a volume, and the other way around.
    • For 6D tools it's usually possible to mount/define/calibrate is such a way that the tool has a natural "axial, coronal, sagital" direction, for 5D tools it's a bit more difficult (no natural up vector for example).

Slicing: Interaction/Control

  • Manual / Mouse mode (MMM): Mostly used in the planning phase, before image data is registered to the patient, VBS/PBS is most common: Interaction directly in the window/view is much easier to comprehend than gui-controls next the view (e.g. slicers vs. cross-hear interaction (e.g. point in axial-view and update the coronal and sagital view accordingly))
  • Automatic / Tracking mode (ATM): Mostly used during tracking (but MMM is sometimes desirable), TBS is most common in Trondheim, but the surgeons switch all the time.

Slicing: Connections/Synchronization

  • Slices are connected in different ways, and they should be updated accordingly (manually or automatic):
    • The most obvious connections are between the 3(2) orthogonal slices in V/P/T/S-based slicing.
    • But connections also exists between a slice shown in a 2D view and the same slice shown in a 3D scene/view.
    • And last but not least, when multiple (overlapping) volumes (from different modalities) exists, corresponding slices arise, these slices can either be shown in separate (but connected) views or the the multimodal information can be combined into a singel multimod slice/view. Please see DesignChallenges.


Fig. 5: MultiModalSlices: The US image in B can be replaced by C, where MRI data from A is filled around the US data without obscuring the updated map of the target area at the same time as an improved overview is achieved. Use of overlay as an aid to interpretation as well as brain-shift assessment will partly hide the data and should therefore be easy to turn on and off (D).


Fig. 6: Orthogonal multi-modal slices in a 3D scene: A slicer object where each of the three orthogonal planes shows a different MRI-volume (T1, T2, and T2 fused with MRA)


Fig. 7:Orthogonal multi-modal slices in a 3D scene: A slicer object where the two orthogonal planes show different volumes (pre.op. axial MR-slice and coronal intra.op. ultrasound slice blended with corresponding pre.op. MR-slice)


Sliceing: Display

  • Center / Zoom / (Virtual) Offset
    • ToolTip-based
      • Volume-centered:


Fig. 8: Volume-centered display: The "volume-data" is centered in the display, and the indication of the tool-tip moves around when the surgeon moves the surgical tool around.

      • Tool centered:


Fig. 9: Tool-centered display: The indication of the tool tip is centered and the slices "moves around". Useful when zooming in, e.g. around the target area or when localizing a landmark.

    • Offset / Virtual-ToolTip-based


Fig. 10: Our surgeons use the offset feature all the time. In stead of using the tool tip (dot) to extract the slices, a virtual tip (small cross, offset mm in front of the physical tool tip, along the main axis of the tool) is used.

    • Overlay:


Fig. 11: It should be easy to turn on and off different types of overlay on the slices. E.g. an indication of the areas covered by the ultrasound probe with the current dept settings or if a segmented 3D surface model is sliced it should be easy to display these contours on the corresponding image slice.

  • Surgeons' position and orientation relative to the patient (on the OR-table) and the surgical tools.
    • The main objective in IGS is to display a navigation scene that is an accurate picture of the physical space to the surgeon, up in slice image is up to the surgeon, right is right, etc.
    • Displaying a slice in a 2D view can never be 100% correct as seen from the surgeon (how should a slice in line with the line of sight be displayed for example), so we have to make the best of it (approx. "correct" is the best we can achieve.
    • Common practice is: Be consistent and the surgeon will learn to adapt to the display with time...
    • For volume/patient-based slicing it helps to define the patient orientation on the OR-tabel (e.g. using a tracked pointer) if the patient position is not conventional (e.g. on the side).
    • For tool-based slicing it's necessary to define the "surgeons' up vector".



Fig. 12: Possible solution (patient(volume)-based orthogonal slicing) that have not been tried in the OR yet, but was considered very intuitive when tried outside the OR by our surgeons some time ago: Some kind of 3D model of the patient (upper left corner) is rotated until the view looks similar to the patient on the operating table as seen from the surgeons point of view/position (the viewpoint and direction can of course also be defined using a tracked pointer (Tracked3DView)). I.e. the patient is laying on the back, front, left side, right side, etc, and the surgeon must choose to look at the patient from one of the 8 possible octant's. In the 3D display above the patient is laying on the back and the surgeon is looking at the patient from the head towards the feet (S->I), from the nose towards the back of the head (A->P) and from the right to the left (R->L). How should the traditional A(xial), C(oronal) and S(agital) slices be displayed in 2D views in the best possible way? Rotate the slices in the upper left corner (2D slices in 3D view) as little as possible in order to be able to display them in the traditional 2D views, in steps of 90 degrees. For example, the slice in the lower left corner is the most intuitive way to display the coronal slice if the surgeon-patient relationship is as depicted in the upper left corner (i.e. left is left, right is right, up is inferior/foot and down is superior/head). For tool-based slicing similar rules can be applied. The important relationship (for the approx. correct display) in this case is the surgeon-tool relationship. The OR-up vector must be clearly defined and the slices must be displayed continuesly, displaying then in steps of 90 degrees is no longer sufficient. This is a good illustration of the fact that it should be possible to show extracted slices in both 2D and 3D views. There are advantages and disadvantages to both techniques.

Things that comes to our attention as we work along

CT/MR/US-ImageReader/ImageSpatialObject/ImageSpatialObjectRepresentation:

Background: In the name of safety there exists special versions (CT/MR/US) of all these classes. In a typical neurosurgical case for example, both pre. op. CT and MR can be used (sometimes just MR, sometimes just CT and sometimes both modalities).

  • Regarding the intended use of the CT/MR/US-ImageReader:
    • Is the app. user supposed to know the modality of the DICOM data he/she tries to open (the app. developer supplies two menu choices: OpenMR and OpenCT) ?
    • If the user do not have to know in advance (a reasonable requirement), should the app. developer first try, lets say a MRImageReader, and if an error of the right type is generated, try CTImageReader, etc.?
    •  ???
    • A humble suggestion from the Trondheim group: Let the user open whatever he wants, read the first DICOM file, find the right modality from the header, if the found modality correspond to one of the supported IGSTK-readers, instantiate the special DICOM reader (CT/MR/etc.) and load the image data, if not (or if the modality-tag is missing) generate the appropriate error event (all encapsulated in an DICOMImageReader, easy for the user, easy for the app. developer, and as safe as you get it...)
  • Is it really necessary to have special versions of the ImageSpatialObjectRepresentation (in terms of slicing it shouldn't matter whether you work with a MR or CT volume)? This is perfectly ok as long as you work with a SINGLE volume but if you work with multiple volumes from different modalities the app. code will not look very nice (there will be a couple of if-statements in such a app. for example....). In order to make multimodal-imaging/viz. work, at some point CT/MR/US data must be regarded as just a voxel-cube, otherwise everything will become to complicated. It's a good time to start to thing of the data in this way, after the volumes have been read into the system
  • It takes some time to load a large DICOM data set, so it would be nice to have a progress bar, is there a way to have update events pass the IGSTK-layer so that they can drive the progress bar?
  • Non of our data looks good using the components above (+View2D) in the standard way . How is the default window with/center set? DICOM header, Voxel content analysis, etc.? When the level/center is changed the data looks ok, and the user should of course be allowed to change these parameters, but shouldn't the default values be close to ok?
Retrieved from "http://public.kitware.com/IGSTKWIKI/index.php/DesignChallenges"
Personal tools
TOOLBOX
LANGUAGES