VTK/Image Rendering Classes

The display of slices of 3D images in VTK is currently much more difficult and much less flexible than it should be. A typical pipeline for displaying an oblique reformat of an image will consist of vtkImageReslice, vtkImageMapToColors, and vtkImageActor, and most VTK novices (in fact even most experts) will have great difficulty sorting through the many settings of these filters to achieve the desired result.

The primary goal of this project is to provide a 3D image mapper that will take care of all the details so that VTK users can display reformats with ease. In order for this to be done, the vtkImageActor must be replaced with a new image prop class that has SetMapper() and SetProperty() methods like those of vtkActor and vtkVolume.

These classes were merged into VTK on March 15, 2011. Major updates were added on June 26, 2011.

Proposal

Oblique view with dark blue background.

I propose a new prop class called vtkImageSlice that will be the image-display equivalent of vtkActor and vtkVolume. This class will have a property class and at least two mapper classes.

• vtkImageSlice
  • vtkImageStack (subclass)
  • vtkImageActor (subclass, for backwards compatibility only)
• vtkImageProperty
• vtkImageMapper3D
  • vtkImageResliceMapper (subclass)
  • vtkImageSliceMapper (subclass)

In addition to the new classes, the vtkInteractorStyleImage will be modified so that it has a "3D mode" for interacting with 3D images.

vtkImageSlice

Sagittal view with dark blue background.

Unlike vtkImageActor, the vtkImageSlice class will have a very simple interface. It will inherit all the vtkProp3D methods, but its only new methods will be SetMapper() and SetProperty().

The existing vtkLODProp3D class will be modified so that it can make an LOD from a vtkImageProperty and vtkImageMapper3D. This will allow vtkImageSlice to be part of an LOD, which was impossible with vtkImageActor. In addition, the VTK pickers will be modified to use vtkImageSlice in place of vtkImageActor (note that vtkImageActor will still be supported, since it will be a subclass).

• SetMapper(vtkImageMapper3D *mapper)
• SetProperty(vtkImageProperty *property)

vtkImageStack

The vtkImageStack is a collection of images that are meant to be rendered on top of one another, similar to the concept of layers in Photoshop. The blending of the layers is controlled by the opacity and by the alpha component of the pixels, and the ordering of the layers is controlled by the LayerNumber stored in the vtkImageProperty. This class also manages the rendering of the images to ensure that the depth buffer values written by each image do not interfere with the rendering of other images in the stack.

• AddImage(vtkImageSlice *image) - add an image to the stack
• RemoveImage(vtkImageSlice *image) - remove an image
• SetActiveLayer(int layer) - set currently active layer

The layering is controlled by the LayerNumber, in preference to the order in which images are added to the stack. The SetActiveLayer() method can be used to control which image will respond to picking. The GetMapper() and GetProperty() methods will also return the mapper/property of the currently active layer, and the VTK interactors will preferentially interact with the active layer.

vtkImageProperty

Axial perspective view with dark blue background.

The property will control the image display parameters.

• SetInterpolationType(int type)
• SetColorWindow(double w)
• SetColorLevel(double l)
• SetLookupTable(vtkScalarsToColors *table)
• UseLookupTableScalarRangeOn() - default Off
• SetOpacity(double opacity)
• SetAmbient(double c) - default 1.0
• SetDiffuse(double c) - default 0.0
• SetLayerNumber(int n) - only used by vtkImageStack

Interpolation types will be Nearest, Linear, and Cubic.

The lookup table is optional. If no lookup table is given, then the window/level will still be applied: single-component data will be displayed as grayscale, and multi-component data will be displayed as color. If a lookup table is provided, the VectorMode of the lookup table can be used to control how multi-component data will be displayed, i.e. by component, by magnitude, or as colors.

Use ColorWindow=255.0/ColorLevel=127.5 with no lookup table to pass RGB or 8-bit grayscale data directly to the screen.

vtkImageMapper3D

This is the base class for 3D image mappers.

• SetInput()
• SetInputConnection()
• BorderOn() - show the whole image, instead of cutting the border pixels in half
• SliceAtFocalPointOn() - automatically choose the slice at the camera focal point
• SliceFacesCameraOn() - automatically choose a slice orientation that faces the camera
• vtkPlane *GetSlicePlane() - get the plane that defines the slice (in world coords)
• UpdateInformation() - update the SlicePlane (do this before using the SlicePlane)

vtkImageResliceMapper

A mapper that can do orthogonal or oblique reformats. Usually this will be used with SliceFacesCameraOn() as part of a 2D viewer, but this class can be used equally well to display an image as part of a 3D scene. In general, use of this mapper is preferred over the less-capable vtkImageSliceMapper because it provides higher-quality interpolation for image scaling and because it is much more efficient for large images, in terms of memory use and speed.

The interface consists of these methods:

• SetSlicePlane(vtkPlane *plane) - set the slice that defines the plane (in world coords)
• SliceAtFocalPointOn() - automatically set slice position to the focal point
• SliceFacesCameraOn() - automatically set slice orientation to face the camera
• SetSlabThickness(double) - set the thickness for slab views
• SetSlabTypeToMean/Max() - set the slab blend mode
• ResampleToScreenPixelsOn() - high-quality mode (on by default)
• SetImageSampleFactor(int) - oversample the image, use factor of 2 for high-quality obliques
• AutoAdjustImageQualityOn() - automatically switch to faster interactive rendering

In its default mode of ResampleToScreenPixelsOn(), this mapper uses vtkImageReslice to reslice the image, and creates a 2D texture as large as the viewport, where all texels outside the image bounds are transparent. This means that the class uses vtkImageReslice, and not the GPU, to perform the interpolation. Although it is not as fast as a pure GPU implementation would be, it has the advantage of being able to deal efficiently with very large images since the texture is always 2D and will never be larger than the viewport, and it provides a very high-quality rendering.

vtkImageSliceMapper

A mapper for x, y, or z slices, i.e. no obliques. It is intended to replace vtkImageActor, which uses it internally. Do no use this mapper if you need to display large images, it will attempt to load the entire image (or slice) as a texture and the rendering will be very slow, always use vtkImageResliceMapper for large images.

• SetSliceNumber(int slice)
• GetSliceNumberMin() - calls UpdateInformation() internally
• GetSliceNumberMax() - calls UpdateInformation() internally
• SetOrientationToX()
• SetOrientationToY()
• SetOrientationToZ()
• SliceAtFocalPointOn() - automatically set slice number to get slice closest to camera focal point
• SliceFacesCameraOn() - automatically set slice orientation so that it faces the camera
• CroppingOn() - default is Off
• SetCroppingRegion(int extent[6])

When given axially-oriented images or 2D images that are 8-bit greyscale or color, this mapper will load the image directly into a texture. It also provides hardware-accelerated bicubic interpolation on systems that support it (which means any GPU released since around 2003).

vtkInteractorStyleImage

This interactor style will be modified so that it can be used for 3D image reslicing. The following methods will be added:

• SetInteractionModeToImage2D()
• SetInteractionModeToImage3D()

When the mode is set to 3D, the following bindings will be present:

• Shift-LeftButton - rotate the camera, i.e. do oblique slicing

In both the 2D and 3D modes, the following new bindings will be present:

• Shift-RightButton - move the focal point in and out, i.e. scroll through slices
• X - sagittal view
• Y - coronal view
• Z - axial view

These keys will change the position and the view-up of the camera. Exactly what view orientations will create the desired ax/cor/sag views will depend on the coordinate system used for the image data. Because of this, the direction cosines for the X, Y, and Z orientations can be set manually with the following methods:

• SetXRightVector(double vector[3])
• SetXUpVector(double vector[3])
• ditto for Y and Z

It is your responsibility to set these view vectors correctly according to the coordinate system that you are using for your images (LPS, RAS, left-is-left, left-is-right, etcetera). The default settings are for RAS, left-is-left. For more information on image coordinate systems, see Orientation.

You can also set the LeftVector and UpVector directly:

• SetImageOrientation(double horizontal[3], double vertical[3]);

A note on window/level: vtkInteractorStyleImage will automatically find the property object of the image that is being displayed and modify it. In the old version of vtkInteractorStyle image, the user had to add observers for window/level. This will no longer be necessary.

Known Bugs

Z-buffer coincidence

When rendering images on top of each other for alpha-blending, if the images do not have exactly the same bounds, then some pixels of the top image might not draw as a result of the limited precision of the vertex pipeline and of the depth buffer. The class vtkImageStack takes care of these coincidence issues, you should always use it when you want to overlay images.

Depth peeling and translucent rendering

The new mappers always render in the Opaque pass, mainly to avoid interaction with DepthPeeling. When DepthPeeling is turned on, per-pixel alpha is turned off during the Translucent pass, which would interfere with image layering. As a result, depth peeling cannot be used to mix translucent images with other translucent geometry.

Wish List

Border pixel clipping

In the current implementation of vtkImageViewer2 the image border pixels are clipped to the half. It would be nice to have an option to show whole border pixels.

done.

The BorderOn() option in vtkImageMapper3D eliminates the border pixel clipping. But for the vtkImageSliceMapper this option only works properly on OpenGL drivers that have GL_CLAMP_TO_EDGE. Some very old GPUs only support the older GL_CLAMP mode which will blend the border with the "border color", which is black by default.

Layer numbers

The order in which images are rendered (and hence composited) depends on the order in which they are added to the renderer. It is tedious for the application to have to remove/re-add all image props whenever the layers have to be re-ordered. If the vtkImageProperty had a "LayerNumber" ivar then re-ordering would be easier. For this to work, image-specific code would have to be added to the renderer.

done.

Checkerboarding

For comparing images, it is very nice to have a "checkerboard" option that will use a checkerboard stencil when compositing the images.

done.

Fast path for vtkImageResliceMapper

The vtkImageResliceMapper always uses vtkImageReslice to interpolate the image at the screen resolution, which means that any pan/zoom/rotation requires a re-interpolation of the image and reloading of the texture. It would be nice if it had a "fast" mode that used vtkImageReslice to do a 3D interpolation to extract an oblique slice, but then used the GPU to interpolate the slice (as a texture) to the screen resolution. This sort of two-stage interpolation results in some artifacts, but the quality would be sufficient for most purposes and the speed would be much faster. This is, in fact, how most people have been displaying images with vtkImageReslice for years.

done.

Slab Projections

It is very common to use "thick slice" averaging to clean up noisy images, or to use MIP slabs when viewing blood vessels.

done. Only vtkImageResliceMapper, not vtkImageSliceMapper.

Shader programs for compositing

Anyone who uses Photoshop or The Gimp will be familiar with the concept of "layers" and the myriad ways that layers can be composited. It would be very nice if the same could be done when displaying images in VTK. It would have to be handled by vtkImageStack, either via software or via multitexturing.

Likelihood: so-so.

Multiple inputs

Instead of having layer numbers, the vtkImageMapper3D could have multiple inputs. It wouldn't solve the ordering problem, but it could make compositing easier. There are several difficulties with this approach, though: the vtkImageProperty would have to be modified so that all of its properties are per-parameter. Also, the picker would not know which dataset to pick from. Overall, layer numbers seem like a better approach than multiple inputs.

Likelihood: low

Render as underlay/overlay

It might be nice to have an option to render images as an underlay to avoid Z-buffer fighting with geometry that is rendered in the same scene. An underlay can be done by rendering the images first (before any geometry) and by not setting the depth buffer while rendering the images. An overlay would be done by rendering the images last and without doing any depth checks.

Likelihood: low

12-bit display

In the medical field, 10-bit and 12-bit greyscale displays are sometimes used. To support these displays, there could be an option to scale the intensity to [0,4095] instead of the usual [0,255]. One problem is that the way to achieve 12-bit display will vary from vendor to vendor.

Likelihood: high if someone is willing to test.

N-Up views

Likewise, it would be easy to take multiple slices of the input image, and reformat them to an NxM grid on a single texture. This would be tricky for picking and getting pixel values, since the portion of the viewport that would typically contain just one image would contain a grid of images instead.

Likelihood: low, easier to just use multiple renderers tiled in a render window.

An image mapper for geometry

It sounds kind of silly, but it could be useful. For example, a FEM could be sliced and displayed as an image. Or a 3D surface contour could be sliced and displayed as an image overlay. The main idea would be to take advantage of the image compositing: there would be no need for the user to make sure that the cutter was set up right and that all the correct depth offset were applied. Instead, the user could just pop the geometry into an image mapper, and the overlay would be perfect every time.

Likelihood: so-so.

Efficiency

The image mappers are designed to achieve the highest quality result, and although they will probably be fast enough to suit almost anyone, they will definitely slow down if displayed in a very large window or if several images are being composited. Hence, it would be nice to have some built-in LOD behaviour similar to the volume mappers. The most obvious speed-up would be to have vtkImageReslice sample the image at a lower resolution, and then have the texture re-interpolate to full resolution. A full-resolution and quarter-resolution texture would always be allocated on the GPU, but only the desired texture would be updated and rendered according to the desired LOD. The interactive rendering would be four times as fast as the high-quality rendering.

Likelihood: low, since the fast path for vtkImageResliceMapper does essentially the same thing.

GPU acceleration

Efficiency could also be improved by using 3D texture maps, but I'm not sure if this would be worthwhile. From a quality perspective, 16-bit textures or floating-point textures would have to be used for medical images. Special fragment programs would be needed for cubic or other high-order interpolation. Lots of GPU memory would be required, particularly since the only time that such high speed would be needed is if many images are being viewed simultaneously. Also, it would be tricky to get it to work correctly with pipeline streaming. The mapper would have to be smart enough to know what parts of the 3D texture would be needed, so that the correct UpdateExtent could be set on the pipeline and the correct SubTexture loaded onto the GPU.

Overall, the current CPU-based implementation is already very fast. A GPU-based image mapper would be unlikely to improve the user experience significantly.

Likelihood: low

Using DrawPixels instead of Texture

The reslice mapper could have an option to render via glDrawPixels. The advantage would be higher performance on non-accelerated system (i.e. Mesa) and minimal use of GPU memory on accelerated systems. The disadvantage in this mode is that the image would have to be re-uploaded to the GPU on every render. This would be bad for performance when interacting with multiple images, since all images would have to be uploaded on each render, instead of just the images that had changed.

Likelihood: so-so.

Color Spaces other than RGB(A)

Already vtkScalarsToColors has a VectorMode setting to say what it does with multi-component input. It would be nice to add a vector mode called "INDEPENDENT" where each component is mapped through a separate color table, and then the resulting colors are summed and clamped. There would be a method where you could add multiple child color tables to the main color table, each of which would map a different component.

Another possibility is to just write a vtkScalarsToColors subclass that does -eg- YBR to RGB conversion. The default conversion should follow the CCIR 601 video standard, but options should be provided for other common YBR/YUV definitions as well.

It would also be good to support a BGRA color space, since that is what most hardware uses natively. Any textures that are not BGRA are converted by the OpenGL driver before they are uploaded to the GPU, so it is kind of silly that VTK converts the images to RGBA just to have the driver immediately re-convert them to BGRA.

Likelihood: so-so.

Smart Scaling

The possibility to visualize the line scanner images (e.g. 1 pixel height and 1024 pixel width). For this purpose it would be nice to have direction-dependent scalling. Also the interaction part must take this scaling factors into account accordingly.

Could the author of this request please explain in more detail?

The current implementation of vtkImageViewer2 will not show an 1x1024 image at all. To visualize images like this you need to create a copy with greater dimension (e.g. 128x1024) and repeatedly memcopy the same image line in it. This approach breaks the interaction part, because you need to implement custom pixel picking and so on... I wonder if it would be possible to use direction-dependent scaling, so you don't need to use a copy of the image line and you can work with original data. This approach could be also applied to images like 10x4096, so you can more effectively use the display area and show the whole image at the same time.

Likelihood: ???.