From IGSTK

Discussion

How shall we handle the following two scenarios? (Patrick)

Luis' Comments

Why a new class ?

• Why do we introduce a new class instead of fixing the ObliqueImageRepresentation class ?
  • These two classes seem to have overlapping purposes.
  • Should we deprecate the ObliqueImageRepresentation ?

Patrick: We should keep and fix ObliqueImageRepresentation class, and make a new class for this one

About Goals

• Goals 2 and 4: defining the (slice) image plane as perpendicular to the instrument is not enough to fully define the plane. (see more comments on this in the API discussion section)
• Goal 5: The functionality of the orthogonal view should not be mixed with this class.
  • The concept of "slice number" is not applicable to non-orthogonal views.
  • Many of the function calls will become context dependent.
    • For example, the class would have
      • a RequestSetOrthogonal(vector) method at the same time it has a RequestSetAxialOrientation() method.
      • a RequestSetSliceNumber( n ) method at the same time it has a RequestSetOrthogonal(vector)
  • Mixing the API of this class with the ImageRepresentation for orthogonal view will lead to a conglomerated code plenty of if() statements, with a fragile API and error-prone interface.
  • Consolidating functionality into "swiss-army-knife" classes is a very detrimental for enforcing the quality standards of IGSTK. We have so far used specialized classes, with the purpose of being able to enforce their correct use. For example, we have a specific class for CT-Images, and a specific class for MR-Images. Thanks to that we can enforce the verification of the modality when we read a DICOM file. If we have used a generic Image class, such verification would not have been possible.

Patrick:

David:

About Requirements

• Requirement 1:
  • This class should not mix its functionality with the one of the orthogonal views.
  • It should be rephrased just as: The ability to do Oblique reslicing according the observed SpatialObject position and orientation.

• Requirement 2:
  • In practice, since this class will be attached to a tracked object, the geometrical definition of the image plane should be expressed in terms of an igstkTransform object (Translation Vector + Rotation Versor) and the direction cosines of a given image.
  • WARNING: Note that VTK does not provide directly the concept of image orientation. IGSTK code will have to map the world coordinates of the tracked object through the direction cosines of the image, before generating the vectors that are passed internally to the vtkImageReslice filter.
  • Patrick: The itkOrientedImage has taken care of the image direction cosines right?
  • ITK direction cosine
  • David: the ITK orientation info is lost when the data goes into VTK. When I use vtkImageReslice, I use the ResliceTransform to describe image orientation/position and ResliceAxes to define the slice orientation/position.
• Requirement 3:
  • The camera position and orientation must be expressed in terms of an igstkTransform
• Requirement 4:
  • Must address how to manage the case of multiple ImageSliceRepresentation objects being attached to the same View.
    • Should we enforce that only one ImageSliceRepresentation can be attached to a View ?
    • Should we enforce that only one ImageSliceRepresentation can drive the camera of a View ?
  • Timing and synchronization is an issue here. Currently, the View's pulse generator drives the updates of the view and gives an opportunity to all the Representation objects to update themselves. Presumably, in that pass, the ImageSliceRepresentation object will get its new position from its tracked spatial object, and will invoke the CameraPositionModified. We must make sure that the View class captures and processes this event before the call to Render().
    • The event CameraPositionModified has a misleading name, since it indicates an accomplished fact. Such event name would make sense if it the event was thrown from the View class after setting the camera to the new position. The event thrown from the ImageSliceRepresentation should rather be called ImageSliceModifiedEvent or something along those lines.
  • Patrick: This is very good point!
• Requirement 5:
  • Image blending capabilities should not be included in this class.
  • If blending is needed, it should be performed by a filter, and its output should be passed to the ImageSliceRepresentation.
  • David: Except that the images might not have the same DICOM orientation, so blending first requires one of the images to be resampled before blending, and then resampled again for oblique slicing.
  • We must avoid overloading classes with miscellaneous functionalities. If we add blending, we should ask why we don't add smoothing capabilities, and segmentation capabilities, and edge detection capabilities... until the ImageSliceRepresentation class will look more like an application than like a component of a toolkit.
  • Good components should follow the motto: Do one thing, Do it right

About Assumptions

• Assumption 1 and 2:
  • If this class is so closely associated to the notion of a needle, maybe we should spell out that fact and make it: ImageNeedleSliceRepresentation, and also introduce a Needle class, so that all these assumptions can easily be enforced.
  • Reuse of code could be achieved by having a base class ImageSliceRepresentation factorizing the code that is not "needle-assumption-specific".
  • Classes with a generic API, whose proper behavior relies on the application developers being aware of certain assumptions are a HUGE safety-risk, and should not be allowed in IGSTK.
• Assumption 3:
  • The geometrical setup of this assumption should be spelled out more specifically.
    • the Z axis here seem to relate to the instrument coordinate system, not to the image coordinate system. This should be clarified in terms of the sequence of transforms used by the Tracker, where the instrument coordinate system relates to the final calibration of the instrument.
    • The UP camera vector must also be defined more specifically.

Patrick: Very interesting point. According to my experience, needle from different company has different local coordinate system. Some define the tip to be -Z direction, some define it to be +Z. How to bridge between this ImageSliceRepresentation class and SpatialObject class?

About Questions

• Question 1: we should ask a surgeon for help here, to list potential surgical instruments
  • Guide wires
  • Catheters
  • Gamma knife
  • Ultrasound probes + video will have a very similar context:
    • Displaying a 2D image in a plane whose orientation and position are defined by a tracked object.
• Question 2: the current implementation is the appropriate approach to maintain scene consistency
  • The same vtkMatrix4x4 is used for
    • Reslicing
    • Controlling the actor holding the image to be displayed
    • Setting the camera in the View class
  • This will make sure that the display is still scene centered and not view-centered, which is an important requirement given that we will have other objects in scene, and we will have multiple views in the application looking at the same scene.
• Additional Question
  • Maybe we should have a specific View class that derives from the generic igstkView class and is intended for synchronizing its camera with events coming from a tracked object. In this way, the other View classes can still enforce their behaviors.

Patrick: I agree

About Issues for Discussion

Issue 1: Class naming

• This class does a lot more than just showing a slice of an image. It probably should include the work "Tracked" somewhere
• InstrumentDrivenImageSliceRepresentation ?
• TrackedSliceImageRepresentation ?
• This also ties up to the option of having an associated specialied View class: TrackedCameraView ?

Issue 2: Deal with assumption

• It was discussed above

Issue 3: State Machine

• This class should not provide an API for orthogonal slices. (reasons were provided above).
• There shouldn't be a RequesetSetProbeTransform(), instead there should be a RequestAttachProbe() method. The timing and transform interactions should be as follows:
  • The representation class will receive a TimeStamp-to-render from the View class.
  • With that TimeStamp, this class will request the Probe for the Transform that is most appropriate for that TimeStamp
  • The probe will get this Transform from its TrackerTool
    • Note that it may turned out to be an expired transform, in which case the ImageSliceView should decide how to indicate to the surgeon that we don't know where the Needle is at this point. (blinking, color change, hiding the slice...)
    • If the transform is valid, the ImageSliceRepresentation will
      • Use it for the vtkImageReslice filter, and
      • It will MAP IT through the image direction cosines before passing it to the vtkActor holding the image slice.
      • The mapped transform will be packaged into the SlicePositionModifiedEvent and throw it for the TrackedCameraView to catch it.
      • The View will catch the event, and set the new camera parameters before calling the VTK Render() method.
  • The point is: if the ImageSliceRepresentation follows a tracked object, then, manually setting the ProbeTransform should not be an option, otherwise, application developers may override the information provided by a tracker, which is a clear safety hazard. If the slice is to be tracked, then a tracker should be the only official source of information regarding the transform that defines the slice.
• The state machine diagram should reflect that fact that this class is not ready to become active until
  • It has a 3D image connected as input
  • It has an spatial object attached to it. (This is presumably a tracked spatial object, but it doesn't have to be. The ImageSliceRepresentation just need to know that it can get a transform from that SpatialObject).

Issue 4: Observe SpatialObject and observed by View

• The ImageSliceRepresentation will observe the spatial object. However:
  • The TransformModifed events from the SpatialObject are not thrown in response to tracker interactions.
  • Trackers store the transforms in TrackerTools.
  • Tracked spatial objects get their transform by directly querying the TrackerTool to which they are attached (they have pointer to m_TrackerTool).
• The ImageSliceRepresentation is requested by the View to do
  • RequestUpdateRepresentation( renderTime );
  • RequestUpdatePosition( renderTime );
  • In the processing methods associated to these two requests, the ImageSliceRepresentation will query its position from the attached spatial object (the needle, not the ImageSpatialObject).and at that point it will throw the event containing its new position, so that the specialized View class can catch the event and update its camera position before calling Render() in its vtkRenderWindowInteractor.
• The Specialized View class only needs a specialized version of the method
  • View::RequestAddObject( ObjectRepresentation* pointer )
  • The specialized version should expect a ImageSliceRepresentation. for example:
  • View::RequestSetCameraTrackedRepresentation( ObjectRepresentation* pointer )
  • Note that the method should be "Set" instead of "Add", because a View class camera can only follow the transform of a single representation object.
  • The state machine of the Specialized view class should ignore the calls from the RequestResetCamera() method, or the specialized view class should overload this method with an empty implementation.

Issue 5: APIs: Provide methods

• This class SHOULD NOT re-implement the functionality of orthogonal-slice image representation classes.
  • Therefore, it shouldn't have the enums for Axial, Sagittal, and Coronal.
• None of the following methods should exist:
  • RequestSetProbeTransform()
  • RequestSetProbeVectorAndPosition()
  • RequestSetResliceMatrix()
  • Because the application developer should not manually set the transform of this class. The transform should be taken from the SpatialObject representing the surgical instrument, in this case, the surgical Needle (tracked or not tracked). The ImageSliceRepresentation should simply get the transform from its associated SpatialObject.
• Just to clarify.
  • The ImageSliceRepresentation has pointers to two Spatial Objects:
    • First: an ImageSpatialObject (the 3D image to be sliced)
    • Second: a SpatialObject associated to the surgical instrument (tracked or not)
  • The transform to be used by the ImageSliceRepresentation should be taken from the second spatial object, the one that represents the surgical instrument.
• The Surgeon-Patient relationship
  • Should only be used for defining the UP vector of the camera.
  • It is not clear whether Left/Right is enough. Neurosurgeons will usually be in-front of the patient's head.
  • This is a high safety hazard, because
    • The position of the surgeon will only be known with certainty once the intervention starts.
    • The position of the surgeon may change during the intervention
    • If the position is wrong, the image will be displayed up-side-down, and it will be very misleading
    • If the position changes at run-time, then the GUI of the application should allow the clinician to change this setting, and therefore it will be prone to human error in the setting, (setting it wrong, or forgetting to change it if the surgeon changes position).

About Specific Methods

• void RequestSetImageSpatialObject( const ImageSpatialObjectType * iso) OK
• void RequestSetSliceOrientation( SliceOrientationType orientation ) Should be removed
  • It pertains to the API of the orthogonal image representation class
• void RequestSetSurgeonPosition( SurgeonPositionType position ) Requires more thinking
  • The specific use of this information should be reviewed.
  • Steps should be taken to ensure that the setting is consistent with the surgery room setup
• void RequestSetOpacity( double opacity ) OK
• void RequestSetWindowLevel( double w, double l ) OK
• void RequestAttachToSpatialObject( SpatialObjectType * so ) OK
• void RequestSetSliceNumber( unsigned int slice) Should be removed
  • It pertains to the API of the orthogonal image representation class
• void RequestSetProbeTransform( TransformType trasform) Should be removed Safety Hazard.
  • Transform should only be taken from the attached SpatialObject.
• void RequestSetProbeVectorAndPosition( double * probeVector, double * probePosition) Should be removed Safety Hazard.
  • Transform should only be taken from the attached SpatialObject.
  • This method is VERY dangerous. It provides an interface based on unsafe raw pointers.
• void RequestSetResliceMatrix( vtkMatrix4x4 * resliceaxes) Should be removed Safety Hazard.
  • Transform should only be taken from the attached SpatialObject.
  • This method is VERY dangerous. It provides an interface based on an unsafe raw pointer.
  • There is not way to enforce that the Matrix has a valid transform, since it is a VTK class that presumably the application developer will setup. A vtkMatrix4x4 can define Affine transforms (while we should only accept rigid transforms), it can also define prespective projections and reflections, all of which are safety hazards.
• vtkCamera * RequestReslice() Should be removed Safety Hazard.
  • IGSTK methods MUST NOT expose VTK classes.
  • If an application developer gain access to the vtkCamera pointer, from it, she/he can pull pointers to all the VTK visualization pipeline, and then all the effort put in encapsulating IGSTK will be lost.
  • Naming was inappropriate: Why ReqestingReslice() should return a camera ?
  • Patrick: This is a quick hack for current implementation, it should not be in the API

Just To emphasize:

IGSTK classes MUST NOT:

• Expose their internal VTK classes
• Expose their internal ITK classes
• Have methods that take raw pointers to non IGSTK objects as arguments

About Internal Structure

• Why does this class has a pointer to a vtkCamera ?
  • Did it took that pointer from the View class ?
    • the View class should not expose its vtkCamera
  • This pointer should not exist in the ImageSliceRepresentation class

Issue 6: Need to define the standard view position and reslice method

• We need input from surgeons here, before we discuss the technical details of a potential implemention

Issue 7: Opacity issue in the VTK pipe line

• Karthik (from Kitware) provided an example on how to manage transparency in the slice.
• The limitation of transparency is that
  • We can not have more than two transparent objects in the same scene
  • This is an OpenGL issue (not even a VTK one), rendering order is important for transparent objects.
• Patrick are you referring here to a different issue ?

Issue 8: Camera reset issue

• The camera settings should be entirely defined by the tracked object
  • In this case the surgical needle
  • There shouldn't be any need for a CameraReset

Note however, that we need to add a couple of parameters more in order to fully specify the camera settings:

• Camera distance to the Focal Point
• Zoom factor of the camera

There should be methods in the API for setting these two values. Presumably, they could be called:

• RequestSetDistanceFromCameraToFocalPoint( double value )
• RequestSetCameraZoomFactor( double value ) (where value must be > 0 )

We also need to define the image grid parameters of the image slice that results from the resampling of the image. This includes:

• Number of pixels along X
• Number of pixels along Y
• Pixel spacing along X
• Pixel spacing along Y

The Transform only defines the mathematical plane that cuts the 3D volume.

Additional methods could be

• void RequestSetSliceExtentInPixels( unsigned int width, unsigned int height )
• void RequestSetSliceSpacing( double spacingX, double spacingY )

Issue 9: Projecting point or line to this resliced plane.

This shouldn't be necessary.

The scene centered approach of IGSTK provides that the camera rendering implicitly performs the projections on a given plane.

We should review the use cases that motivate this functionality. There are probably better ways to achieve the same results.

Should the cases be compelling, then the implementation of such functionality should be made in a separate representation class. Otherwise, this class will have an attack of feature creep

Issue 10: Need testing program for this class

We need testing programs for ALL IGSTK classes.  :-)

Full coverage testing is a requirement of IGSTK.

It shouldn't be necessary to mention the need for testing programs here.

• Patrick: This is for my own reminder :p

Hazard Analysis

Probe outside of image volume

• This shouldn't be a problem. The output slice should simply be blank. Of course, we should setup a test in order to verify that this is the behavior.

SpatialObject is not updating it’s position

• This should be managed in the same way that all other Representation objects do
  • The visibility of the Slice should be changed when the Transform returned by the tracked spatial object is an expired Transform.
    • This could be done by blinking, changing opacity, or color, or by simply hiding the slice.

Changing between different modes

• Switch between manual reslicing to auto reslicing
  • There shouldn't be any manual slicing mode. That's a safety hazard.
  • Only automatic reslicing should be provided, and it will be controlled by the transforms of the SpatialObject

Change of surgeon standing position

• This feature requires more thinking
• We need more specific input from surgeons
• It should consider the following cases
  • What to do if surgeon change position with respect to the patient
  • What if the patient is in the supine position ?
  • What if the surgeon is a neurosurgeon ?
  • How to label this in the display
  • How to deal with the eventual need of changing this setting from the GUI of an application
    • If the setting is available on the GUI then: How to ensure that it is set and kept up to date

Required changes to IGSTK

Open the Camera access to igstkView

NO WAY !

IGSTK classes must not expose any of their internal VTK or ITK components.

There is no need to expose the Camera:

• The camera position and orientation can be set by the specialized view class based only on
  • The transform provided by the tracked spatial object
  • The camera distance to the focal point

igstkView observe the ImageSliceRepresentation

This is indeed a necessary change.

However it shouldn't be implemented at the level of the igstkView class.

Instead, a new specialized class, deriving from the igstkView class, should be introduced.

This new specialized View, will be aware of the need for observing the ImageSliceRepresentation.

Additional comments about the originally Proposed API

• Off-Axial view is geometrically ill defined (at least in the wiki page).
  • The image plane must be defined by one point (in this case the needle tip) and
    • two vectors parallel to the plane OR
    • one vector perpendicular to the plane and one vector indicating the camera Up direction
  • From the description of the Off-Axial view it seems that
    • The plane is defined as the plane parallel to the Left-Right axis and parallel to the axis of the instrument (the needle in this case).
    • The specification of the Up camera vector seem to be missing
• Off-Sagittal view is geometrically ill defined (at least in the wiki page).
  • The image plane must be defined by one point (in this case the needle tip) and
    • two vectors parallel to the plane OR
    • one vector perpendicular to the plane and one vector indicating the camera Up direction
  • From the description of the Off-Sagittal view it seems that
    • The plane is defined as the plane parallel to the Superior-Inferior axis and parallel to the axis of the instrument (the needle in this case).
    • The specification of the Up camera vector seem to be missing
• The Perpendicular View is geometrically ill defined (at least in the wiki page).
  • The image plane is be defined by one point (in this case the needle tip) and
  • The image plane is defined as perpendicular to a user-provided vector
  • The specification of the Up camera vector seem to be missing
• Since the Coordinate System of IGSTK is NOT the coordinate system of any of the potentially loaded images, this class must make sure that it takes into account the origin, spacing and orientation (direction cosines) of the image that it is re-slicing.

[Comments] from the Trondheim group:

Appendix

How I define the reslice plane

itk::Vector< double, 3 >       vx, vy, vn, v;
 if( m_SliceOrientation == OffAxial)
   {
   // Calculate the reslice axes
   vx.Fill( 0.0 );
   vx[0] = 1; 
   vy[0] = m_ProbeVector[0];
   vy[1] = m_ProbeVector[1];
   vy[2] = m_ProbeVector[2];
   if ( fabs(vx*vy) < 1e-9 )
     {
     // FIXME: need to handle this situation too
     std::cerr<< "The two vectors are parrelell" << std::endl;
     }
   vn = itk::CrossProduct( vx, vy ); 
   vy = itk::CrossProduct( vn, vx );
   v.Fill( 0.0 );
   v[0] = 1;
   if (vx*v<0)
   {
     vx *= -1;
   }
   v.Fill(0.0);
   v[1] = 1;
   if (vy*v<0)
   {
     vy *= -1;
   }
   v.Fill( 0.0 );
   v[2] = 1;
   if (vn*v<0)
   {
     vn *= -1;
   }
   vx.Normalize();
   vy.Normalize();
   vn.Normalize();
   }
 else if ( m_SliceOrientation == Perpendicular)
   {
       // Calculate the reslice axes
   vx.Fill( 0.0 );
   vx[2] = -1; 
   vn[0] = m_ProbeVector[0];
   vn[1] = m_ProbeVector[1];
   vn[2] = m_ProbeVector[2];
   if ( fabs(vx*vn) < 1e-9 )
     {
     // FIXME: need to handle this situation too
     std::cerr<< "The two vectors are parrelell" << std::endl;
     }
   vy = itk::CrossProduct( vn, vx ); 
   vx = itk::CrossProduct( vy, vn );
   v.Fill( 0.0 );
   v[2] = -1;
   if (vx*v<0)
   {
     vx *= -1;
   }
   v.Fill(0.0);
   v[0] = -1;
   if (vy*v<0)
   {
     vy *= -1;
   }
   vx.Normalize();
   vy.Normalize();
   vn.Normalize();
   }
 else if( m_SliceOrientation == OffSagittal)
   {
   // Calculate the reslice axes
   vy.Fill( 0.0 );
   vy[2] = -1; 
   vx[0] = m_ProbeVector[0];
   vx[1] = m_ProbeVector[1];
   vx[2] = m_ProbeVector[2];
   if ( fabs(vx*vy) < 1e-9 )
     {
     // FIXME: need to handle this situation too
     std::cerr<< "The two vectors are parrelell" << std::endl;
     }
   vn = itk::CrossProduct( vx, vy ); 
   vx = itk::CrossProduct( vy, vn );
   v.Fill( 0.0 );
   v[1] = 1;
   if (vx*v<0)
   {
     vx *= -1;
   }
   v.Fill(0.0);
   v[0] = -1;
   if (vn*v<0)
   {
     vn *= -1;
   }
   vx.Normalize();
   vy.Normalize();
   vn.Normalize();
   }
 // set the reslice axes  
 m_ResliceAxes->Identity();
 for ( int i = 0; i < 3; i++ )
   {
   m_ResliceAxes->SetElement(i, 0, vx[i] );
   m_ResliceAxes->SetElement(i, 1, vy[i] );
   m_ResliceAxes->SetElement(i, 2, vn[i] );
   m_ResliceAxes->SetElement(i, 3, m_ProbePosition[i]);
   }
 m_ImageReslice->SetResliceAxes( m_ResliceAxes );
 m_ImageReslice->SetOutputSpacing( 1,1,1 );
 m_ImageReslice->SetOutputOrigin( -m_SliceSize/2, -m_SliceSize/2, 0 );
 m_ImageReslice->SetOutputExtent(1,int(m_SliceSize+0.5),1,int(m_SliceSize+0.5),0,0);
 m_Actor->SetUserMatrix(m_ResliceAxes);
 //Setting up the camera position
 double focalPoint[3];
 double position[3];
 for ( int i = 0; i<3; i++ )
   {
   focalPoint[i] = m_ProbePosition[i];
   position[i] = m_ProbePosition[i];
   }
 for ( int i = 0; i<3; i++ )
   {
   position[i] -= m_CameraDistance * vn[i];
   }
 vtkCamera   * camera = vtkCamera::New();
 camera->SetViewUp( -vy[0], -vy[1], -vy[2] );
 camera->SetFocalPoint( focalPoint );
 //camera->SetParallelScale( 0.8 );
 //camera->Zoom( 2 );
 camera->SetPosition( position );