ImageSliceRepresentation

From IGSTK

Introduction

This component is developed according to the requirements of surgeon and is based on previous experiences with case studies.

Current Issues

1. igstkImageRepresentation (Orthogonal views), reflecting the tip position of the needle, is very hard to get the orientation information out of the display with the needle overlay. Thus, when the surgeon maneuver the needle, it’s hard to correlate the screen display with the physical movement, and it very difficult to align needle with planed path to target tumor
2. igstkImageRepresentation doesn’t have a mechanism to do the automatic reslicing. (In needle biopsy application, we have a pulse generator to do the reslicing)
3. igstkObliqueImageRepresentation class doesn’t function correctly, and is not easy to use

Goals

1. be able to reslice image in arbitrary way; be able to represent and navigate in the image volume better. ( A video showing how easy to target the tumor using the new slice representation )
2. be able to display image slice in plane or perpendicular to the surgical tool, which gives the surgeon more information on the scene, and better guidance to manipulate the needle (Refer to the appendix on Off-Axial, Off-Sagittal, Perpendicular view).
3. be able to design the class in a way similar to the SpatialObject-Tracker binding architecture, allow the ImageRepresentation object to be attached to a SpatialObject, so it will handle the reslicing internally according to the SpatialObject position.
4. be able to adjusting the camera position to always look perpendicular to the image plane.
5. it will be better we can implement the orthogonal slice representation in the same class, and enable to reslicing using slice number.

Design Discussion

Requirements

1. The ability to do Oblique/Orthogonal reslicing according the observed SpatialObject position, or respond to standard reslicing request with slice index
2. Able to automatically change the camera view angle and position in the View class to best display the image.
   • Need to develop and confirm with surgeon on a series of standard views
3. Be able to observe SpatialObject’s TransformModified event and do the automatic reslicing internally according to some predefined reslicing mode, and also calculate the new camera position
4. generate a CameraPositionModified event, and it will be observed by igstkView class when a slice representation class is added to a view.
   • A design similar to vtkFollower, instead, it’s the camera following the actor, always position itself to look at the front of a actor
5. Be able to incorporate image blend in the future

Assumptions

Questions

1. Is needle probe a sufficient representation for all surgical instruments? What if the principle axes changes?
2. In this implementation, we set the reslicing matrix to the vtkActor (Fig.2: Internal Classes Collaboration), no texture mapping is used, so the vtkActor is being displayed in the exact position where the slice is being extracted.
   • Advantages 1: No need to calculate the complicated transform matrix to map this to some standard plane (as in texture mapping)
   • Advantages 2: The image slice representation actor is being placed in the correct position in the surgical scene, so it will have more correct relative position with respect to other objects in the display window
   • Disadvantages: Disadvantage is that, we need to constantly changing the camera position to look perpendicular to the vtkImageActor.

Issues for Discussion

1. Naming of this class
2. Deal with the assumptions
3. State Machine
4. Observe SpatialObject and observed by View
5. APIs: Provide all of the following?
   • RequestSetProbeTransform
   • RequestSetProbeVectorAndPosition
   • RequestSetResliceMatrix
6. Need to define the standard view position and reslice method with respect to the surgeon standing position (Left or Right side of patient), which is more intuitive and comfortable to surgeon
7. Opacity issue in the VTK pipe line
8. Camera reset issue
9. Projecting point or line to this resliced plane.
10. Need testing program for this class

Hazard Analysis

Required changes to IGSTK

1. Open the Camera access to igstkView, and igstkView class is able to observe the view changes from the ImageSliceRepresentation class

Implementation Details

Use Case Diagram

Fig.1: Use Case Diagram, External Collaboration

From lower-right of the image

1. SpatialObject (a needle probe) get updated transform from Tracker
2. ImageSliceRepresentation class observe the TransformModifiedEvent sent out by SpatialObject, and perform reslicing internally
3. The resliced image is passed to View class for rendering, View class also observe the CameraModifiedEvent, and adjust its camera position accordingly

Collaboration Diagram

Fig.2: Internal Classes Collaboration

From the igstkTransform, the ImageSliceRepresentation class will calculate a reslice matrix (vtkMatrix4x4) according to the orientation mode, this matrix will also be used to setup the actor position, it will also calculate the new camera position according to the reslice matrix.

Interface design

1. Enum defines some standard views (Axial, Sagittal, Coronal, OffAxial, OffSagittal, Perpendicular)
2. Enum defines Surgeon-Patient position relations ??? (Patient’s Lef, Patient’s Right)

void RequestSetImageSpatialObject( const ImageSpatialObjectType * iso)
void RequestSetSliceOrientation( SliceOrientationType orientation )
void RequestSetSurgeonPosition( SurgeonPositionType position )
void RequestSetOpacity( double opacity )
void RequestSetWindowLevel( double w, double l )
void RequestAttachToSpatialObject( SpatialObjectType * so )
void RequestSetSliceNumber( unsigned int slice)
void RequestSetProbeTransform( TransformType trasform)
void RequestSetProbeVectorAndPosition( double * probeVector, double * probePosition)
void RequestSetResliceMatrix( vtkMatrix4x4 * resliceaxes)
vtkCamera * RequestReslice();

Structure Diagram

Fig.3: Structure Diagram

State Machine Diagram

This state machine diagram needs lots of work, the class involves too many parallel states.

Fig.4: State Machine Diagram

Current Development

ImageReslice Spatial Object and Representation classes:

• igstkImageReslicePlaneSpatialObject < igstkImageSpatialObject >

• igstkImageResliceSpatialObjectRepresentation < igstkImageSpatialObject >

Classes driven by the resliced plane:

• igstkMeshResliceSpatialObject and igstkMeshResliceSpatialObjectRepresentation: this class uses vtkCutter to cut the igstkMeshSpatialObject set as input, and employs the resliced plane as implicit function.

• igstkCrossHairSpatialObject and igstkCrossHairRepresentation: so far, this class only works on orthogonal orientations (OrientationType::Axial, OrientationType::Sagittal, OrientationType::Coronal). It sets a vertical and horizontal line (vtkLineSource, vtkTubeFilter) on top of the resliced plane. We use red, green, and blue for the axes.

• igstkToolProjectionSpatialObject and igstkToolProjectionRepresentation: this class sets a line (vtkLineSource, vtkTubeFilter) representing the projection of the tool onto the resliced plane. In the case of resliced planes in which the tool long axis lays on, the length of projected tool will remain fixed and equal to the value set by RequestSetSize() method in igstkToolProjectionSpatialObject

List of related Events:

• VTKCameraModifiedEvent: this event is sent by igstkImageResliceSpatialObjectRepresentation and should be observed by a class derived from igstkView2D (e.g. igstkView2DPlaneDriven). We need to listen to this event because we want the camera in the View to follow the reslicing plane. Now I am listening to this event from the application and sending the camera parameters to the view (igstkView::SetCameraViewUp, igstkView::SetCameraFocalPoint)

• VTKPlaneModifiedEvent: this event is sent by igstkImageReslicePlaneSpatialObject every time the reslicing plane is updated

• ImageBoundsEvent: this payloaded event can be used to get image bounds from VTKImageData

• IntegerBoundsEvent: this payloaded event can be used to get orthogonal bounds (int) from VTKImageData

ImageReslice Reslicing Types: Orthogonal (Axial, Sagittal, Coronal), OffOrthogonal (Off-Axial, Off-Sagittal, Off-Coronal), Oblique (PlaneOrientationWithXAxesNormal, PlaneOrientationWithYAxesNormal, PlaneOrientationWithZAxesNormal)

TODO list:

• fix the extension of the resliced plane according to its intersection with the image bounds. Note that the intersection can be a polygon with eight points.
• move code from igstkImageResliceSpatialObjectRepresentation to igstkImageReslicePlaneSpatialObject with respect to the calculation of the pixel dimension in the resliced plane.
• in igstkMeshResliceSpatialObjectRepresentation make sure that the contour is always visible when is rendered. For the moment we are using vtkTuber but something lighter should be used instead (e.g. line width property)

 Collaboration Diagram between classes that depend on igstkImageReslicePlaneSpatialObject

 Event Flow Diagram

 CrossHairs and ToolProjection objects in the views

 MeshReslice example: an OffSagittal View and intersected contours

New design need:

we run into the need of having Plane Reslice component working without an image Something like this:

 MeshReslice component working without image

proposed change: set a generic Spatial Object to the Reslice Plane instead of ImageSpatialObject

Appendix

Illustration of different Views

Legend:

• Red line with arrow – Needle, the probe
• Blue dash-line plane – Reslice plane
• Green dash line – Reslice plane’s rotating axes, which always passes though the probe tip position

Fig.5: OffAxial View. This is called the Off-Axial View mode. 

As the needle/probe pivot in the Cranial-Caudal (head-foot) direction, the reslice plane will rotate with the needle, sweeping though the image volume. The resulting display shows the needle always in-plane with the image slice. You can see large angular movement of the needle in the Off-Sagittal view window. Demo Video

 Fig.6: OffSagittal View

As the needle/probe pivot in the patient's Left-Right direction, the reslice plane will rotate with the needle, sweeping though the image volume. The resulting display shows the needle always in-plane with the image slice. You can see large angular movement of the needle in the Off-Axial view window. Demo Video

 Fig.7: Perpendicular View

This view mode can be set to be perpendicular to a probe, or any other vectors. In our case, this plane is being set to be perpendicular to the planed path. And it will reslice according the tip position of the needle. Demo Video

Background about vtkImageReslice

 m_ResliceAxes->Identity();
 for ( int i = 0; i < 3; i++ )
   {
   m_ResliceAxes->SetElement(i, 0, vx[i] );
   m_ResliceAxes->SetElement(i, 1, vy[i] );
   m_ResliceAxes->SetElement(i, 2, vn[i] );
   m_ResliceAxes->SetElement(i, 3, m_ProbePosition[i]);
   }
  
 m_ImageReslice->SetResliceAxes( m_ResliceAxes );
 m_ImageReslice->SetOutputSpacing( 1,1,1 );
 m_ImageReslice->SetOutputOrigin( -m_SliceSize/2, -m_SliceSize/2, 0 );
 m_ImageReslice->SetOutputDimensionality( 2 );
 m_ImageReslice->SetOutputExtent(1,int(m_SliceSize+0.5),1,int(m_SliceSize+0.5),0,0);

Fig.8. vtkImageReslice

vtkImageReslice needs four input parameters to setup the ResliceAxes, they are: X-axis, Y-axis, plane normal, and a point on the plane.
What ever point you set in the ResliceAxes matrix, it will be transformed to the (0,0,0) point in the output image space.
In this case, we use the tip position of the probe, and it will be mapped to the (0,0,0) point in the resliced image. As previously mentioned in Assumption 2, we will place the resliced image plane centered around this point, so we need to set the origin accordingly, taking into consideration of the spacing and image size. The grey margin is the automatic padding, due to the OutputExtent is larger than the available image volume.

Preliminary results

Fig.9: User Interface for the Navigation System

This is the GUI developed using the new ImageSliceRepresentation class.
In the bottom-right window, 'Perpendicular to planed path':

• Red sphere is the tip of the needle
• Green sphere is the target
• Yellow sphere is the hub of the needle, or the end of the needle

If the needle is perfectly aligned with the planed path, these there sphere should overlaid on each other.

This give us a powerful and easy way to do the targeting:

1. Move the needle, so that the red sphere is on top of the green sphere. Now the needle tip is on the planed path
2. Keep the tip of the needle steady, and pivot the needle, so that the yellow sphere is on top of the green and red sphere. Now the needle is aligned with the planed path
3. Advance the needle and keep monitoring the display for needle position and orientation information and make corrections accordingly.

Refer to the 'Targeting' part of the Demo Video

Video Demo

A Youtube Video showing the navigation GUI