VideoGrabberComp

From IGSTK

Contents 1 VideoGrabber component: Introduction 2 The Role of the VideoGrabber Component in IGSTK 3 Structure of the VideoGrabber Component 3.1 Buffering 4 State Machine 5 Component Interface 5.1 Interface Methods 5.2 Events 6 Implementation suggestions 6.1 Timing 7 Background information 8 Time schedule

VideoGrabber component: Introduction

The component is needed for real-time image data input in IGSTK platform. This initial proposal is modelled after the Tracker component in IGSTK.( igstkTracker, igstkSerialCommunication, see also: vtkVideoSource )

Some application areas

• Updating the navigation scene with real-time information
• Use of new real-time modalities -> Multimodal information
  • Ultrasound Strain
  • Ultrasound Blood Flow Imaging
  • Endoscopy
  •  :
• real-time information during guiding (fluoroscopy, etc.)
• Temporal and spatial calibration
• registration based on real time images

Requirements (Draft)

• Import of real time video without noticeable delay.
• Import via FireWire, USB-2 or internal PCI grabber card.
• Make sure the component works with hardware used by the IGSTK partners.
• Grab video stream or grab single image.
• Handle multiple input streams.
• Timetag support (must be able to syncronise with position data from Tracker component).
• Support for crossplatform development. Goal is cross platform support (but not necessarily all platforms at once).
• Support for different video input standards (analog support in first version) :
  • PAL
  • NTSC
  • VGA
  • Medical radiology monitors (Higher resolution than PAL, NTSC).
  • DVI
  • HD (different resolutions : 480p, 720p, 1080p, and 1080i).
  • MPEG-2 (Video in DICOM).
• Support for different video output formats :
  • Grayscale 8 bits.
  • YCrCb 16 bits
  • RGB 24 bits.
  • RGBA 32 bits.
• Support buffering of videostream for later processing (ultrasound 3D reconstruction etc.)
• Support rendering of video directly to graphics memory.
• Should be able to set clipping or padding of video output from the component.

Use Case The typical use of this component :

• Open a specific grabbing device (adress, port, device driver name). The videograbber then instantiate a grabber for the current platform.
• Set parameters for the video output. Dimensions, clip, padding, pixelformat. Set destination for the videodata - Direct to memory adress or internal buffer. Set size of internal buffer. Maybe the videograbber should inform the user about its capabilities ?
• When device is open and all video output parameters have been set, the component reaches state GrabberActive.

The user can now either (repeatedly) call GrabOneFrame() or start continous grabbing with StartGrabbing(). After StartGrabbing() VideoGrabber will reach state Grabbing and can no longer accept GrabOneFrame().

The Role of the VideoGrabber Component in IGSTK

Fig.1: System overview - Real-time Imaging

The focus of the VideoGrabber component is the grabbing/import of an analog 2D real-time image stream. See the above figure to get an overview of the VideoGrabber's part with respect to other possible real-time imaging modalities.

The VideoGrabber component will make the IGSTK platform able to import live video streams for use by the navigation application. These video streams can originate from different sources such as endoscopes, microscopes, X-ray /fluoroscopy and ultrasound scanners.

Structure of the VideoGrabber Component

 

Fig.2: Overview of the VideoGrabber component.

The VideoGrabber interfaces to the external video sources by a Grabber object. This Object will have platform dependent code like Quicktime for Mac. The code of the VideoGrabber that uses the Grabber object will run in a separate thread, while the rest of the VideoGrabber code (including the state machine) will run within the main IGSTK application thread.

 

Fig.3: VideoGrabber Class Diagram

Buffering

 

Fig.4: The Video Buffer used by the VideoGrabber

The Video Buffer will be implemeneted like a ring buffer and be able to store a limited number of video frames. Objects needing access to the whole buffer or parts of it (like the last frame), should access the data through pointers to prevent unnecessary memory consumption.

While the grabber is in continuous mode, data consistency must be upheld by using mutexes or other synchronization methods. For the grabber, it is important to minimize the risk that the grabber looses an image due to the buffer being locked by a reading thread. Therefore double-buffering should be used in continuous mode. This requires the allocation of two identical buffers when initializing grabbing, one read-only (R) and one write-only (W). The grabber continuously writes to the W-buffer, while any thread trying to read an image should read from the R-buffer. After a grab is completed, pointers to the buffers swap, while enforcing multi-threading constraints using mutexes or other suiting methods.

State Machine

 

Fig.5: State Machine diagram

• Major States

1. Idle : Initial state.
2. GrabberReady : Grabber ready to use.
3. GrabberActive : Grabber activated. Allows RequestGrabOneFrame() calls.
4. Grabbing : Grabbing video either to buffer, texture or both. A separate thread handles the updates.

• Transitional states

1. AttemptingToInitializeState : Attempting to initialize Videograbber, and selects and initializes platform dependent Grabber object.
2. AttemptingToActivateGrabber
3. AttemptingToStartGrabbing
4. AttemptingToStopGrabbing
5. AttemptingToDeactiveteGrabber
6. AttemptingToGrabOneFrame

Component Interface

Interface Methods

1. RequestSetVideoBufferSize()
2. RequestGetVideoBufferSize()
3. RequestGetVideoBufferPointer() : Return pointer to the allocated videobuffer.
4. RequestGetVideoFrameNo() : Return current video frame number.
5. RequestGetNumberOfVideoFrames() : Return number of video frames in the buffer.
6. RequestGetTimeTag(int frameNo) : Get timetag for indicated videoframe.
7. RequestSetWantedFramerate() : Set the framerate.
8. RequestGetFramerate() : Reports the actual framerate.
9. RequestSetVideoOutputFormat() : Set video format. Grayscale, color, opacity ..
10. RequestGetVideoOutputFormat()
11. RequestSetVideoOutputClipRectangle() : Clip the video image before storing in the buffer.
12. RequestGetVideoOutputClipRectangle()
13. RequestSetVideoOutputPadding() : Pad the video output image before storing in the buffer. (To make image dimension a power of 2 etc.)
14. RequestGetVideoOutputPadding()
15. RequestSetGrabbingMode() : Grab continuously (write frames directly to texture memory), grab to buffer or both.
16. RequestGetGrabbingMode()
17. RequestSetTexturePointer(Texture* ptr) : Target for video frames when using continuous grabbing.
18. RequestOpen()
19. RequestClose()
20. RequestInitialize() : Initialize VideoGrabber and choose and initialize platform specific Grabber object.
21. RequestStartGrabbing() : Enter the grabbing state and start grabbing.
22. RequestStopGrabbing() : Stop the grabbing.
23. RequestGrabOneFrame() : Return one video image.

Events

1. VideoGrabberOpenEvent : Generated when a call to RequestOpen() was successful.
2. VideoGrabberOpenErrorEvent : Generated when a call to RequestOpen() failed.
3. VideoGrabberCloseEvent : Generated when a call to RequestClose() was successful.
4. VideoGrabberCloseErrorEvent : Generated when a call to RequestClose() failed.
5. VideoGrabberInitializeEvent : Generated when a call to RequestInitialize() was successful.
6. VideoGrabberInitializeErrorEvent : Generated when a call to RequestInitialize() failed.
7. VideoGrabberStartGrabbingEvent : Generated when a call to RequestStartGrabbing() was successful.
8. VideoGrabberStartGrabbingErrorEvent : Generated when a call to RequestStartGrabbing() failed.
9. VideoGrabberStopGrabbingEvent : Generated when a call to RequestStopGrabbing() was successful.
10. VideoGrabberStopGrabbingErrorEvent : Generated when a call to RequestStopGrabbing() failed.
11. ...

Implementation suggestions

To render a real-time video signal we need a new SpatialObject (type VideoSpatialObject) . This object will contain the geometric information for the ultrasound sector (or endoscope/microscope image etc.) and will be attached to a VideoGrabber. It will get its image information from a VideoGrabber instead of a ImageReader. The geometric information must be set (and changed during operation) according to the ultrasound scanner settings. When a new videoframe is received by the VideoGrabber an event will be generated and received by the observing SpatialObject. A timestamp will be set for the videoframe (provided by the video capture driver or generated by igstk). The SpatialObject will not have a copy of the videoimage but the SpatialObjectRepresentation will have a reference to the texture object being the destination for the videostream from the VideoGrabber. This will keep the video data in graphics memory keeping the load off the CPU. It will also enable videoprocessing using the processor on the graphics board (GPU). QuickTime and possibly also DirectShow have the possibility of setting up pipelines of filters for videoprocessing. These filters can be implemented on the GPU for better performance. Comments
In the first version of the videograbber, video data will be stored in a RAM buffer and not rendered directly to a texture in a OpenGL scene. We need to do tests to investigate the performance gain by doing a render-to-texture solution.

Timing

When the View is being updated the SpatialObject will fetch the transform from the TrackerTool corresponding to the timetag for the last videoframe. Note that this doesn't nescessarily need to be the latest transform for the TrackerTool since the video stream and the tracker will have different update frequencies.
Issue for discussion
Maybe we should let the videostream trigger the updating of the View, so that for every new video frame a View update is done ?

Background information

Crossplatform support :

• Mac:
  • API: QuickTime
  • Driver: QuickTime-compatible driver
• Windows:
  • API: DirectX/DirectShow (IS:Glossary, see also WfW)
  • Driver: WDM Stream Class driver (IS:Glossary, Wikipedia, MSDN )
• Linux:
  • API: Video4Linux (V4L) (see also Unicap)
  • Driver: V4L driver

Hardware used by the IGSTK partners :

• SINTEF:
  • HW: Video-to-FireWire converter (DFG/1394-1e) from The Imaging Source
  • Mac:
    • API: QuickTime
    • Driver: OSX driver (universal)
  • Windows:
    • API: DirectShow
    • Driver: WDM Stream Class driver
  • Linux:
    • API: Unicap
    • Driver:
• etc.

Relevant links:

• Imaging Source website: Glossary, What is: WDM, DirectX
• Potentially relevant:
  • DICOM WG24 (Surgery): Minutes
  • The Khronos Group: OpenML
• etc.

Time schedule

 

Fig.6: Implementation plan