Talk:DesignChallenges

From IGSTK

Jump to: navigation, search

Luis, David, I really appreciate your comments, hopefully the discussion will continue and it will be possible to conclude at the end. Frankl

Contents

Coordinate systems / Transforms / Registration

  • The goal of supporting all the cases of single/multiple pre-operative/intra-operative datasets fits well into the expected use of IGSTK Luis. I'm not sure that the current IGSTK code base support this statement 100% Luis. My point is that this should be build into the system in an elegant way. Frankl
  • IGSTK is designed using a scene-centered representation,
    • All datasets can be registered to that common frame.
    • In the current state of IGSTK, volume datasets are read and displayed at the locations and orientations indicated by their DICOM files.
    • When registration is neecessary, as an initialization of the surgical intervention, then the image registration classes should be assimilated to "readers" in the sense that they are image producers.
    • If image registration was to be performed on a continuous basis, eg. US to MRI registration, then the registration class is to be assimilated to a "tracker" in the sense that it provides new transforms with a certain regularity.
    • Currently none, of the cases described above is implemented in IGSTK.
      • The fundamental components, however, are available in ITK and VTK.
      • IGSTK currently has a igstkMR3DImageToUS3DImageRegistration class. This class should be used inside a new MRI3DUSImageTracker class, that will produce transforms to be set in the US image spatial object that is passed to the representation object that displays the US in the scene Luis. Yes I agree (this is typical menu-choice today but a MRI3DUSImageTracker might be needed in the future, but without the use of Tracker in the name...). I hope that you mean that it's the transform of the MR spatial object that should be updated Luis. One question regarding the igstkMR3DImageToUS3DImageRegistration class. I have been trying to fined the time to look into this class for a long time. Does it actually do what it says? That would be great because this is hard to do in a robust and accurate manner (we have been working with this for a long time so we know...). And one more thing: I do see the need for specialized version of this class. Maybe it should be even more specialized, taking things as the body part imaged and whether tissue or angio data is provided into account. Frankl.

This section of the "Design Challenges" page is not yet very clear. Please let me know if my interpretations below are correct:Dgobbi (It's meant for the IGS expert David..., basically it says that in order to have a general framework that can deal with "all cases" (which we should have) the concept of volume transforms are introduced. In the special case only one pre.op DICOM dataset is used everything stays the same (but a general framework is used)). If other more elegant alternatives exists we should describe them and test the concepts against the cases listed. Details and IGSTK adoptions should of course be taken into account. Hopefully the discussion will make it clearer. Frankl)

  1. It must be made clear how the "volume coordinate system" is defined with respect to the voxel data. Since we are primarily using DICOM images, it seems to me that the volume coordinates must be identical to DICOM coordinates, i.e. must take not only the voxel spacing but also the DICOM Position and Orientation information into account. Dgobbi 07:31, 10 May 2007 (EDT). Yes Frankl. One question: In the special case that two acquisitions from the same modality (e.g. MR-T1 and MRA, different spacing, size, etc.) are generated in the same session and we assume that patient do not move between will IGSTK and the way it handles the DICOM info take care of the registration automatically? Frankl No, IGSTK does not check whether image volumes were acquired in the same session. Dgobbi
  2. For the case of a single Pre-op data set, it is stated that patient coordinates will equate to volume coordinates, and the VT will be the patient registration transform. From this, I understand that VT is the transformation that goes from tracker coordinates (relative to the reference) to volume coords.Dgobbi 07:51, 10 May 2007 (EDT). Your assumptions are right David Frankl

To help me understand this a little better, we can consider a case involving both a CT and an MR which have been registered together, giving a volume-to-volume transformation (ct_VT_mri). After these images are loaded into IGSTK, we would proceed with a fiducial registration of the tracker to the MRI. This would give us the VT for the MRI. Then, using the ct_VT_mri, we would calculate the VT for the CT. Is this correct?Dgobbi 08:04, 10 May 2007 (EDT). The mechanism for updating the VTs will be crucial, it might be worth adding a new section to the Design Challenges page specifically for this. Dgobbi 09:36, 10 May 2007 (EDT)

Comments: Frankl

  • I think it's very informative to illustrate potential design choices in terms of concrete scenarios like this, we should do it whenever possible.
  • I think that the scenario shows the basic idea behind the "general volume-transform framework" a place to put/accumulate all this update information at the same time as the IGS app. do not need to know about all the updates throughout the operation, it just uses the concatenated volume-transform to go from tracking coordinates to volume/image coordinates (the update transformations should of course be "saved" in addition to have the accumulated volume transform. In addition (and very important) this framework can extended to deal with intraop. data (e.g ultrasound) and anatomical shift updates (see below).
  • But the (relatively simple(just two pre.op. datasets)) example also illustrates the first fundamental change that must be made to IGSTK in order for this to work elegantly: the tracker component can no longer deliver the tool locations in patient(volume) coordinates, i.e. it must deliver the tool (tip) locations in tracking coordinates (relative to a reference frame of course) to (each) ImageSpatialObject?, that has it's own VT, telling the position of the tool tip in terms of volume coordinates (with or without updates), that in tern is used by a SliceImageRep. for example???.

In the past I have always defined my "surgical coordinate system", to be the same as the primary MR image coords. That is why the tracker is the way that it is, it had to report tool positions in that coordinate system. When I had an ultrasound or a CT, I stored the registration transform of those images relative to the MRI. For proper multimodal support, I agree that this is not the best way since there is not always a "primary image volume". I will change my way of thinking. We can have an intraoperative registration transform stored in each image spatial object, and none stored with the tracker. Dgobbi

  • Scenario
    • Pre.op.:
      • All available pre.op. data (n) is read (VT = DICOM header transf.?)

Please, Frank, no abbreviations. I first read that as "DICOM header transferred." Dgobbi

      • If more then one volume is available the surgeon would like to view (planning before pas.reg.) all the data in a common frame of reference so a V2V reg. is run between a master (e.g. the volume that is registered to the patient) and the rest of the volumes (n-1) giving n-1 update transforms that are applied to the respective volume transforms (see CT/MR scenario above). (Reslicing is of course an alternative)

We will need to define an event that will be generated when the update of the transforms is needed. The V2V transform will also have to be stored somewhere, either in ImageSpatialObject or perhaps in a new object. Reslicing is bad, storing the transforms and applying them when viewing the data is of course the best. Dgobbi

    • Intra. op.:
      • The master volume (e.g.) is registered to the patient and the VT basically becomes equal to the pas.reg.transf. For the other volumes the their VT must be updated with the pas.reg.
      • Intra.op. data are acquired: No problem they get their own VT telling the location of the volume relative to the reference frame / tracking space.
      • The surgeon would like to update the pre.op. data to the match the current patient anatomy given by the intra.op. data so (multi.mod.) V2V reg. algorithm is run (based on vessels for example) and the found transform. can be applied to all pre.op. (potentially, this could be a non-rigid reg.)
      • The last two points can be repeated over and over again (for the purpose of resection control for example). Saving the update transforms (so that the situation in the OR at any time can be recreated) and concatenating the old VT with the update to give the new VT for the different volumes.
      • The extreme case is, as Luis mentioned, when hight res. real-time 3D intra.op. data becomes available the updating process can be run continuously (not possible today I think, but s

Logging the VTs as they change is an excellent idea. Be very careful about "concatenating the old VT with the update". When you get into continuous updates, numerical errors will start to accumulate. A VT should be created in one of only two ways: 1) stored directly as a result of an image-to-tracker registration, or 2) created by concatenating another volume's VT with a V2V transformation. Creating a new VT by multiplying the old VT by an update is bad news. Dgobbi

    • Post. op.:
      • If postoperative data (e.g. MR/CT) is available the update process could be taken one step further, updating both pre- and intra-op. data to the post. op. situation for comparison and post.op. control.

V2V and VT information will both be present for most volumes. When doing the updates, they should always be done using the V2V information since the V2V almost always the "original data." Dgobbi

So the basic question is: Is there anybody else besides me that thinks that this is a good idea (alternatively, do other solutions exists) and if so how can this be done in an elegant way within the IGSTK framework??? Do you agree with me that an early and simplified conclusion is that every ImageSpatialObj. should be associated with it's own VT (or IT (Image Transform) to use the right ITK/IGSTK notation) and that the tracker comp. should be adapted to this design (i.e. give the the tool locations in tracker (relative the reference) coordinates??? Frankl

We will also have to consider mesh data (surfaces). The VT should be part of the SpatialObject base class, not just for images. For tracked spatial objects (i.e. tools) the VT will be the tracker's transform for that tool and will be updated continuously. Dgobbi

Update efficiency / SurgicalSpace-ImageSpace delay / "Temporal safety"

A "push" driven update is not a good idea, in my opinion. IGSTK is not real-time and there are no guarantees that one "push" will be done before the next one starts. The advantages of polling are that (1) it can be done quite fast, polling at 20Hz should be more than sufficient (2) polling is less complex and leads to a simpler, more robust design.Dgobbi 08:21, 10 May 2007 (EDT)
David, Luis: It would be nice if you could elaborate a little bit about why (overall) polling is faster and simpler/more robust. Frankl

I agree with David here. We also have to point out again that discussion about efficiency and performance must be based on measurements taken from repeatable experiments. As the complexity of the system increases it is not trivial to judge how the temporal performance is going to evolve. Before suggesting to modify the structure of the event passing mechanisms we should measure their behavior in realistic cases under controlled circumstances. If such experiments reveal that there are performance issues, they will also provide the opportunity for evaluating and comparing alternative approaches, such as a push event architecture. Luis. 21:36, 16 May 2007 (EDT)
Yes, we agree, again, that decisions should be based on measurements, whenever possible. Discussions however..., still, it's probably wise not use to much time on this right now if the common opinion is that potential benefits are questionable, but it's also an educational dimension to this as stated above, so please be patient with us...Frankl

There are benefits to both, but remember that "pull" is the basis to the VTK and ITK pipelines, so we can expect that IGSTK programmers will have more experience doing things that way. Dgobbi

Comments: Frankl

  • It should be emphasized that the poll vs. push discussion isn't completely "black or white", different levels exists:
    • completely pull (close to the way IGSTK works today)
    • completely push
    • something in between: the suggestion was that instead of letting a separate pulse generator steer the update process for each view the tracker comp. could send update events (push) to all views (poll) when new tracking data arrive.
  • The background for the suggestion is a concern that can be illustrated with the following scenario:
    • Let's say that we have a standard four quadrant-display (2x2, 1x3D + 3x2D), and each view runs a pulse generator (PG) at 30Hz. In addition the tracker and videograbber runns runs their own PGs as 30 Hz.
    • So each second 4*30=120 + 2*30=60 = 180 events are generated and processed (in addition to the synchronization events between the views, etc.). There will be a lot of activity behind the scenes. And if the surgeon is busy with other things (that don't require navigation, i.e. black screens) all this activity will be there anyway...
  • There might be a better solutions???, but is this a valid concern??? To give a preliminary answer myself: with current computer power I don't think that this will emerge as a problem for simple IGS application but when more complex features are added I wouldn't be surprised if it was (time and measurements will show)

The updating mechanisms in IGSTK is something that we should leave alone for now. It is a complex subject and involves many (or all) of the components. Once we have specific applications to test against, we can re-think the design if we find any flaws that must be fixed. Dgobbi

Multimodal Visualization

The overlapping volume problem is one that has to be solved in VTK, not IGSTK. There are two ways to do it:

  1. Combine the two volumes into a single 3D texture before uploading to the GPU. This is easy to do, but requires that you resample and crop one of the images to match the other.
  2. Use advanced GPU programming to render the volumes together. Perhaps with the new "depth peeling" in VTK, this would not be too difficult to achieve. The advantage over "pre-compositing" the volumes into a single texture is that you get higher quality visualization, but rendering will be twice as slow.

One thing to note about "pre-compositing" into a single texture rather than doing the compositing with pixel shaders, is that pre-compositing only works well if the volumes are of similar extent and resolution.Dgobbi 08:29, 10 May 2007 (EDT)

Comments: Frankl

  • Many good points here David.
  • Together with slicing and surface/polygon rendering I think that volume-rendering has it's natural (visualization) place in IGS/IGT (se below), in that order, i.e. start with a good slicing comp. and do the vol.rend.comp. last.
  • I think that the natural path would be to start with a "IGSTK wrapping" (state machine, restricted to needed functionality, etc.) of VTK's current volume-rendering framework and have a plan to implement the design and implement the future comp. for GPU-based volume-rendering in IGS (probably in VTK as stated). * As stated previously I think that such a comp. should be able to handle multiple (not just two) overlapping volumes with different size, spacing, position, orientation etc. with the speed necessary for IGS. It might be worth looking into GPU-based ray casting.
  • We have already implemented such a comp. based on 2D textures (at present however the volumes can not be rotated relative to each other) so it should be possible.

Turning on the depth peeling in VTK cvs might be all that you would need to do in order to allow the 2D textures to be rotated relative to each other. Would it be possible for you to test this? Dgobbi

Volume Rendering Assumption

The assumption that volume rendering is a good way to approach vascular visualization at high speed is one that need to gather some supporting data. Volume rendering is a form of fuzzy segmentation. As such, it may be more efficient to actually run a fuzzy segmentation technique on the vascular data and then generate polygonal representations from it for the purpose of rendering.

When rendering a volume dataset that represents vasculature, most of the rays go through empty space. Therefore, the rendering approach should perform some level of smart leaping in order to avoid spending time on the many voxels that do not contain blood vessels.

Luis 21:56 May 2007 (EDT)

Comments: Frankl

  • Many good points here, but I strongly believe that a volume rendering comp. is a natural extension of IGSTK (when other more important things have been taken care of)
  • Vascular data was used as en example here, but I believe that volume-rendering also has a natural place in tissue visualization, however this requires good clipping features (overlapping voxel values).
  • As long at we want to visualize tissue borders (the case in IGS) the geometric/triangle/surface/polygon vs. volume rendering debate will continue, because both techniques can show similar things. Let me try to say a few words about why I think that vi ALSO need volume rendering functionality in IGSTK:
    • pre.op.(good time to segment, manual seg. possible in order to improve the quality) MR/CT (relatively easy) vs. intra.op.(no time between availability and display) Ultrasound (more difficult to segment). It's no coincidence that vol.rend. is used in systems for real-time 3D ultrasound. On the other hand, USAngion (doppler) is almost as easy to segment as CTA/MRA making it more suitable for surf. rend.
    • Detail and closeness to "gold standard":
      • Slicing is the "gold standard" in IGS because the surgeon can't get any closer to his medical image map then that: it is only interpreted through the window level/center (good presets can be defined) and the detail is as good as you get it.
      • Direct volume rendering is it's 3D equivalent: the voxel data is only interpreted through a couple of transfer functions (good presets den be defined) and fine details can be shown.
    • Interactivity: changing the transf. func. you immediately see the effect (nice feature, this is normally not the case for surf.rend.)
    • Robustness: Vol.rend. is more robust than surf. rend./segmentation. Vol.rend. never fails, it will always show something and together with clipping it's possible to generate nice visualizations most of the time.
  • Regarding vol.rend. of vascular data:
    • Often only a limit region of the available angio data is interesting (e.g. vessels around a tumor, clipping needed, empty space reduction)
    • In real applications it's often paramount to see vessels that are hardly detected by the image scanners. It's hard to see these vessels with surf.rend. With vol.rend. you only turn the opacity transf.func up or down to see it something disappear or show up.
    • Seen from the comp. developer the empty space problem could justify a specialization: TissueVolRend and AngiVolRend, but seen from the app. dev. this increases the complexity of the program.
  • Some sentences from an old article: Surface versus Volume Rendering
    • Although 2D slicing is essential for detailed information interpretation, complex 3D structures must be mentally generated based on a stack of 2D slices. This requires years of experience, and is one of the reasons why research groups have now introduced various 3D display techniques for planning as well as in the OR for surgical guidance. Computers using modern 3D rendering techniques are particularly useful for assessing complex 3D structures like the vascular tree, and for obtaining an overview of important relations between relevant structures (e.g., infiltrating vessels in a tumor). Theoretically, it is possible to apply both volume-rendering and geometric extraction techniques directly to volume data, as well as to segmented data. For practical visualization of 3D MRI and US data, we often found that it was possible to generate nice views by isolating interesting parts by clipping the volume and opacity- classifying the content of the sub-volume. Another important advantage of volume rendering is that both the surface of the object of interest as well as the inner content (e.g., for a tumor with cysts) may be displayed. Geometric rendering of clinically interesting structures is most successful if an intermediate segmentation step is performed first, so that an accurate surface representation can be generated. Although advanced methods for automatic or semi automatic segmentation exist, manual methods must often be used, especially for US data. In many cases, it is also necessary to verify the tumor border, e.g., in a low-grade tumor where it is hard even for an experienced radiologist to delineate the border. Still, promising segmentation methods exist. One example is the deformable models approach, where a template (taken from an anatomical atlas, for example) is deformed to fit new US volumes acquired during the operation. In summary, it is our experience that volume rendering is the most appropriate 3D visualization method for US data, since the generation of a surface representation often requires a segmentation step, which is generally a more demanding task than segmentation based on MRI. In addition, the time available for the additional segmentation step is more limited in the OR than pre- or postoperatively

Comments: Dgobbi

A definite advantage that I think volume rendering has over segmentation is that volume rendering provides more information to the surgeon.

Segmentation means that an algorithm is choosing what the boundaries of a structure are. Vascular segmentation is particular tricky because there are always bits missing where the vessels are too thin or where image contrast is not sufficient.

A surgeon has a much better understanding of the vascular than any algorithm, and given the "fuzziness" of the volume rendering I suspect that a surgeon can do a better job of deciding what is vessel and what isn't. It is just a suspicion, but unless the segmentation can be proven to be perfectly robust, I'd always feel more comfortable leaving the decision to a surgeon rather than trusting the algorithm.

Slicing multiple Volumes

Combining multiple volumes in a single visualization by cutting (slicing) them with a single plane seems to be an appropriate functionality to implement in a single representation class (SpatialObjectRepresentation) connnected to a MultiImageSpatialObject. As you suggested, the MultiImageSpatialObject would be connected to multiple Volumes (igstkImages) each one with a corresponding transform that maps its coordinate system to a common reference.

As discussed earlier:

  • The transforms may be provided by registration classes that behave as Trackers, or
  • The images can be provided by registration classes that behave as image sources.

Slicer 3.0 implements a common slice functionality in which multiple volumes are resliced and their contributions are combined in a single intersecting plane. It might be a good idea to evaluate this representation as a candidate for an implementation of the similar functionality in IGSTK.

Luis 22:06 May 2007 (EDT)

NA-MIC has decided to re-work the Slicer3 functionality since it isn't efficient enough for IGT. I have my own code, but it is very complex. To achieve efficiency, I have different code paths for different situations, plus I incorporated level-of-detail rendering. In the the long run, perhaps this will have to be addressed by designing some better image display classes for VTK. Dgobbi

Comments: Frankl

  • With the MultiImageSpatialObject approach there will probably be data duplication problems, e.g. when the same volume should be displayed alone and together with other volumes at the same time. The one representation connected to multiple ImageSpatialObj. (each with a single volume) seems to be better, but this is not allowed. We should really find a good solution to this problem. Traditionally we have used a "volume manager" for this task (in addition to a view manager for view synchronization).
  • Good point, we should definitely look at Slicer3 (and Davids slicer classes, as well as our in-house slicer)
  • Some sentences from an old article: chapter: Slicing versus 3D Display
    • Displaying slices in separate windows (Fig. 2B, E) with crosshairs overlaid to indicate the current tool tip position, makes it possible to display many slices without obscuring others. The drawback is that it might be difficult to handle all this information that is distributed on many separate windows. We have shown that the slices (one to three) from one to several volumes (acquired pre-, intra- or postoperatively) may be displayed together in one window, making it easier to interpret information. Furthermore, the US image in Figure 5B can be replaced by the one in Figure 5C, where MRI data from Figure 5A is filled in around the target area for an improved overview but does not obscure the updated US data. On the other hand, overlays used to aid US interpretation often hide information (Fig. 5D), and should hence be easy to turn on and off. Still, it is difficult to mentally match images presented in this way with current patient anatomy as seen from the surgeon’s viewpoint, or to understand the orientation of the surgical tools relative to the orthogonal volume slicing. The orientation problem can be solved by integrating the 2D planes in a 3D scene (Fig. 2C, F) and manually rotating the scene until the viewpoint corresponds to the surgeon’s view. This may also be controlled automatically by tracking the head movements of the surgeon in addition to tracking the surgical tools and probes.Potential problems with this approach are that some slices will be partly obscured (Fig. 2C), and that slices almost parallel to the viewing direction will be difficult to see (Fig. 2F). To minimize these problems, only relevant information should be displayed, and it should be easy to turn on and off the different objects in the scene. Furthermore, the orientation of the 3D scene can be tied to the traditional 2D display of the slices so that the information is presented as close to the surgeon’s view as possible (Fig. 5E). Thus, when a surgeon moves the instrument to the left relative to himself, the tracked instrument will also move (approximately) left in the axial and coronal slices, while the sagittal window will display new data, extracted further to the left as seen from the surgeon’s perspective.

Slicing

I agree that, in a surgical application, having distinct objects for "orthogonal" vs. "oblique" slicing is not necessary. It should be enough to have a single "slice" object.Dgobbi 08:38, 10 May 2007 (EDT)

Options

How many classes ?

I disagree with having a single slice object. It will require a messy API in which multiple use-cases will have to be supported, probably by switching modes on and off. It will be cleaner to have specialized objects in which we can enforce restricted behaviors. The "Swiss-army" approach of having classes that provide many different functionalities spells doom for a safety-critical application. Specialization makes possible to define constrains, and defined constrains make possible to enforce safety. Luis 22:15 16 May 2007 (EDT)

Comments:

  • I think that we are at the core of our disagreement here Luis. What you say sound like a good idea seen from the component level, but the consequences on the application level (in terms of complexity: having many objects around, switching between them etc) would be dramatic in order to support the clinical user level in most cases (not just the complex cases), and as user / surgeon requirements should be our guiding star (we have no choice at lest) this will also have consequences in terms of safety for the system as a hold: In order to satisfy the surgeon/user (the app. dev. has no choice) the app. will be very messy and unsafe if build on top off components that are not designed for the real clinical world. And if the toolkit is not easy to use in order to create real apps. the toolkit will not attract potentially interested users/app devs. I will try to elaborate a little bit more about this view later today if I find the time. Frankl


We can have (at least) three types of slice object representation classes, and maybe their associated spatial object that is connected to multiple volumes:

  • Slice that is orthogonal to a single volume
    • It moves at discrete (integer) intervals
    • It has three orthogonal modes
  • Slice that is orthogonal to a single volume
    • It moves in a continous way(maybe by getting positions from a tracker)
    • It has three orthogonal planes
    • It can switch each one of those plane On/Off on request.
  • Slice that has a free moving plane
    • This class may be connected to multiple volumes
    • The plane is defined by a point a normal vector and a UP vector
    • The image grid on the plane is defined by
      • An origin
      • Number of pixels along each dimension
      • Spacing along each dimension
    • This slice object may also have three orthogonal planes to switch on/off. The planes will follow the coordinate system attached to the a moving (e.g. tracked) object.

The free moving plane Slice object can obtain the parameters of its plane (origin, normal and up vector) from a Tracker. In this way, it will be possible to model a Tracked ultrasond probe, for example. Luis 22:20 16 May 2007 (EDT)

Comments:

  • The last alternative is starting to sound like a slicer object that is needed in a surgical application. I do not think that there is a real need for a separate integer class and that orthogonal slicing relative to the volume and orthogonal slicing relative to the tool can be treated in the same object without adding to much complexity. At least, the benefits would out wight the drawbacks (if a real need for such classes should emerge, this would be the special cases).
  • Having three orthogonal planes to switch on/off seems to be an good idea from a synchronization/work load (if all three slices are used moste of the time)/etc. point of view, but this could be more challenging in terms of display/viewing: should a single SliceRepresentation be attached to three different views or should a new "ThreeSubView" object be created. Frankl

In my mind, having separate classes to handle integer versus continuous slicing makes no sense. When the surgeon wants to switch from one situation to another, the application will have to switch all the objects around and re-do a bunch of connections. Handling both cases in a single class is straightforward, you simply have to duplicate most of the states so that the appropriate code is called in each case. There will be a lot of shared code, of course, which can be placed in private methods that are called from the processing methods of both the "indexed" states and the "continuous" states. Dgobbi

Synchronization

Synchronization of multiple slices is quite challenging. There are multiple ways to implement it, and the risk cases are

  • Sending too many update messages and end up triggering redundant renderings
  • Not sending enough messages and loosing the update of a particular view.

One potential way to get around this problem is to take advantage of the scene centered nature of IGSTK. This could be done by having the MultiImageSpatialObject hold the information describing the plane that is intersecting all the volumes. Part of this information may be obtained from a tracker (e.g following an ultra-sound probe). Since the information describing the position of the SpatialObject plane goes to the Tracker tool, and the spatial object query the transfrom values when it is required to do so by the ObjectRepresentation class, this will provide a natural synchronization.

Note that still the view classes as the ones marking the rythm at wich the video refresh updates are being performed.

Luis 22:27 16 May 2007 (EDT)


Slicing Display

The options of Slicing display tie to functionalities of the View classes. In the particular case of the tool-centered representation, the View class must be aware of the position and orientation values that the Tracker reports for the tool in question. The tracker coordinates (tool position and orientation) can then be used for computing/updating the camera parameters of the vtkCamera in the View. Luis 22:32 16 May 2007 (EDT)

The alternative visualization of the ultrasound slice (for example, the boundary representation based on lines) could be a functionality provided by an additional spatial object representation class.

Thing that came along

Image Specializations US/CT/MRI

Yes, the user of the application is supposed to know what type of modality is about to load into the system. As you pointed out, the GUI will probably offer the specific options of Load CT, Load MRI, Load US, etc.

In an image guided surgery intevention a user SHOULD know what type of modality is about to read. This may not be the case in a generic image processinng tool, but it certainly should be the case in an application designed to be used in the surgical room.

A variation on the suggestion by the Trondheim group: Instead of letting the user select any image and then punish her/him with an error message if the image turns out to be the wrong one, we could modify/specialize the GDCMFileNameGenerator class for taking as optional input a modality. In this way, when the list of DICOM files in a directory is presented to the user, only the ones that match the requested modality will be presented in the GUI. In this way, we only present to the users image that already valid options for the image loading process.

Luis 22:32 16 May 2007 (EDT)


I do not see any good reasons for forcing the user to know what kind of data he/she is loading. And it's not practical. Because the DICOM-loader/user will probably not be the surgeon (the only one that might know for sure). In Trondheim we do the job if we are present in the OR, if not, one of the nurses does, in any case this is done before the surgeon is entering the room (i.e. before we have any chance to speak to him), so the modality information most follow the media in order for the user to know, e.g marked CD's (could be wrongly marked...) delivered in the reception by the radiology department or in the file-names of the dataset generated by the DICOM server receiving the data. This is definitely not the case always, and reading the data with an other app. first to find the modality is not a good solution either. I might be exaggerating the problem a bit here, for many procedures only a single pre.op. modality is used (could be different from time to time thought), my point is why not "generalize" when there is little or no penalty to it, after all, for any given patient an increasing amount of image data will be available in the future. Only if the user selects a dataset that is not supported by the IGSTK app. (or the modality tag is missing) a error event is generated which is OK (something have definitely gone wrong, and the user should know). I think that the present implementation contradicts one of the messages from "The Book": safety by design, whenever possible make it hard for the user and app. developer to make mistakes (and deal with it if they do, because they will). I think that the alternative suggestion is very good for the user if everything is OK, but if they are not I would rather have the nurse call the radiology department directly telling them that there is something wrong with the data that was sent (because an IGSTK error message told them so) then calling me saying that there is something wrong with the system because the data will not show up in the file browse even thought he/she knows that the data is on the CD/disk. In terms of the app. and comp. developers I think that the alternative will be more complex. Frankl

Modality Specializations of the Image Representation classes US/CT/MRI

By having image representation classes specialized for different modalities, it is possible to enforce rule on the type of display. For example, in the case of CT we can use the hounsfield units in order to callibrate an appropriate intensity windowing function.

It is true that if we have a Slice representation class that gets information from multiple volumes simultaneously, then there will be a need for an intensity callibration or intenstiy compensation process, that will scale the intensities of the CT, MRI, PET, US...etc. images to match a common scale. Otherwise some image modalities will be preseneted as very bright, while others will appear very dark. Luis 22:45 16 May 2007 (EDT)

The window/level issue is one all visualization software has to deal with, and for each modality there is a suitable rule to apply:

  • CT: data is calibrated, so start with either a "bone" window or "soft tissue" window.
  • US: data is 8-bit, so start by directly mapping to an 8-bit color table.
  • NM and PET: calibrate data, map brightest pixel to the top of the color table.
  • MRI: This one is tricky, but I have a histogram analysis class that does a very good job of providing a suitable window/level.

Dgobbi 19:34, 23 May 2007 (EDT)


Again, this is my main concern with IGSTK. I think that in order to prepare IGSTK for the future we have to generalize whenever possible and specialize whenever necessary, without sacrificing the overall system safety (if not it will be to hard to create real IGS apps. for the OR). I think that this is possible because the functionality that is required in an IGS app. is finite and well defined. MRSliceImageSOR (SpatialObjectRep): No way (SliceImageSOR should be ok), MRImageSO (SpatialObject): no, ImageSO should be ok, contains the image data (itkImage) together with the VT or ImageTransform (IT), a modality tag (=MR), the WC/WW (or this should probably be in the SliceImageSOR, but the values are found when reading the data so I'm not sure, and if the app. should offer other presets this could be based on the modality tag assosiated with each volume), etc., MRImageReader: probably a good idea if hidden from the user and app.dev. inside a DICOMImageReader (the right modality reader is instantiated after the modality is found from the first file for example and supported), MR2USImageRegsitration: god idea. Basically, I think that it's as easy as this, when designing a toolkit for IGS we have to think of the user(surgeon), the application developer and the component developer, both in terms of simplicity and safety, etc, and both the user, app.dev and comp.dev. individually and together.

  • User: Should offer the requested/required functionality (which in most cases are complex), if not the toolkit will have no future.
  • App.dev.: Should be relatively easy to create the apps. requested by the users, ideally it should be as easy as to connect the necessary hight level components and put a GUI on top making the user able to access the functionality offered by the components , if not the toolkit will will not be used (small user community)
  • Comp. dev.: At some level the complexity required by the user must be put, I strongly believe that the components are the right place. However, by carefully designing the core components it should be possible to manage these by a state machine, also the more general comp. when such comp. are needed in order to make it more easy for the app. dev. because simpler more specialized comp. means more complex applications in order to meet the requirements given by the user.

The concept of "code usage" should not be underestimated either, when it comes to a smaller number of more general comp. vs. a larger number of more specialized comp.: The best insurance against comp. errors is that the comp. is used all the time in real apps. and the best insurance for proper use of the comp. is a lot of examples and documentation showing this (it's difficult to have the proper overview if a lot of specialized objects exists). Maintaining a lot of comp., some of which are hardly used, also requires more resources. Frankl

Progress Reporting from DICOM reader

This is a great idea. The implementation will go inside ITK, and maybe we will have to retouch GDCM for this purpose. But, in anycase, it is a good point. We don't want to have the surgeons wondering what the system is doing, and if it is not getting responsive.

Luis 22:53 16 May 2007 (EDT)

Luis, could this be done in a way that would make it easier to implement the same functionality for other cpu-intensive tasks taken from ITK (or VTK), for example image(V2V) registration? Frankl

I agree, all tasks that take more than 5 seconds should have a progress bar. Especially in image-guided surgery, the surgeon might decide it is better to continue the surgery without the software than to wait for an undetermined amount of time, since every extra ten minues of OR time adds to the risk of complications. Dgobbi.

Intensity Windowing Initialization

This is a good point and probably should be entered as a bug. The initial callibration of the intensity values to the display should be based on the information of the image modality and some overall evaluation of the image histogram.

This is a functionality that must be implemented at some point.

Luis 22:54 16 May 2007 (EDT)

One of my employees wrote a VTK class that does this. It finds the 5% and 95% threshold points of the histogram (or whatever thresholds are desired). The method works well, I can contribute this class to VTK. Dgobbi

One more thing: if the WC/WW is set in IGSTK (ImageSliceRep.) to the WC/WW given in the DICOM header (should be the default set in IGSTK I think) the image does not look similar to the same data displayed in several other DICOM viewers (which are similar). Shouldn't the same data displayed with the same WC/WW give similar images in different viewers as long as these viewers use the hole dynamic (voxel value) range? Frankl

Retrieved from "http://public.kitware.com/IGSTKWIKI/index.php/Talk:DesignChallenges"
Personal tools
TOOLBOX
LANGUAGES