From IGSTK

Contents 1 Project Description 1.1 Problem Statement 1.2 Specific Aims 2 Material and Methods 2.1 Phantom 2.2 Data Interface 2.3 Image Calibration 2.4 Tracked Instrument 2.5 Segmentation & Reconstruction 2.6 Surface reconstruction with Kinect 2.7 Visualization 2.8 System Integration 3 Action Items

Project Description

We propose to develop a navigation system that will leverage our previous work in image-guided surgery to provide the laparoscopic surgeon with an augmented reality view of:

• subcutaneous structures such as vessels and tumors, thereby providing “x-ray” vision beyond what can be seen with the laparoscopic camera.
• projection of pre-operative plan on treatment target to provide real time guidance for surgeon.

The system will enable navigation and image guidance for laparoscopic surgery, including the fusion of laparoscopic ultrasound with pre-operative CT and laparoscopic video and deformable modeling of abdominal organs for real-time updates during the procedure.

For Phase I of this project, we will focus on image fusion of ultrasound and laparoscopic video.

Problem Statement

Laparoscopy is performed through small incisions, resulting in loss of free sight, dexterity, and tactile feedback as compared to open surgery. Therefore, one of the main drawbacks with the laparoscopic approach is that the surgeon loses the ability to palpate organs, vessels, and tumors during the procedure. The surgeon has a restricted view inside the body cavity through the video laparoscope. Any information beyond the surface of the organs visualized is lost. As a result of this limitation, surgeons must rely on visual cues from the video monitor rather than tactile feedback to achieve comparable or even improved outcomes as those reported for laparotomy. Because of this limited surface view, the surgeon must mentally correlate the underlying anatomy with this view, often including information from medical images obtained prior to the procedure. Therefore, there is a need for more advanced visualization of the patient’s anatomy and pathology to provide the surgeon with information below the surface view of organs.

Because of the aforementioned limitations, laparoscopic ultrasound is often being used to supply additional anatomic information. Ultrasound is real-time and nonionizing, ideal for intra-operative imaging. However, interpreting ultrasound images and correlating them to laparoscopic view are very challenging, especially when the depth information is lost in the laparoscopic video. Our goal here is to create an intuitive fusion of ultrasound image with laparoscopic video.

Specific Aims

The end goal is to develop a system that will argument subsurface vessel structures in laparoscopic video, specific aims include:

1. Develop a flow phantom for ultrasound imaging, suitable for simulating vessel
2. Develop data interface to acquire ultrasound and laparoscope image
3. Develop ultrasound and laparoscope image calibration method
4. Develop tracked ultrasound and laparoscope instrument and its calibration scheme
5. Develop ultrasound vessel segmentation algorithm and 3D reconstruction algorithm
6. Develop surface model reconstruction method based on Kinect depth camera
7. Develop an augmented reality display to combine 3D vessel structure and organ surface with texture mapping

Material and Methods

Below are proposed solutions based on the specific aims above.

Phantom

1. Georgetown phantom
   • Might need to connect to a pump to create flow
   • Problem: flow perpendicular to US probe, might not work in Doppler mode, unless we tilt the probe

1. Kyoto Kagaku "IOUSFAN" phantom, http://www.kyotokagaku.com/products/detail03/us-3.html
   • very realistic, specially designed for laparoscopic applciations
   • Problem: no vessel flow included

1. Build a custom phantom
   • Flow model with vessel tree for segmentation and registration with color Doppler ultrasound and CT

Data Interface

1. Ultrasound: Terason research API
2. Laparoscope:  ??
3. VideoGrabber - under development, http://public.kitware.com/IGSTKWIKI/index.php/ITKA2D2%2C_Generic_VideoGrabber

Image Calibration

1. Ultrasound: Ziv is developing an automatic algorithm
2. Laparoscope: Raj has some existing code ?

Tracked Instrument

Optical vs EM?

1. Ultrasound probe
2. Laparoscope

Segmentation & Reconstruction

1. Option 1: segment vessel in 2D ultrasound image and then reconstruct 3D vessel structure
2. Option 2: reconstruct 3D ultrasound image from tracked 2D probe and the segment vessel in 3D
   • Freehand 3D ultrasound reconstruction algorithms--a review
   • An Open-Source Solution for Interactive Acquisition, Processing and Transfer of Interventional Ultrasound Images
   • An open-source real-time ultrasound reconstruction system for four-dimensional imaging of moving organs
3. Consider dynamic update of the model

Surface reconstruction with Kinect

1. Patrick has a Kinect and can explore its API
   • TURN KINECT INTO 3D SCANNER EXPLAINED FULL TUTORIAL WITH CODE
   • http://openkinect.org
     • http://openkinect.org/wiki/Imaging_Information
   • Kinect calibration, http://www.ros.org/wiki/kinect_calibration/technical
   • OpenFrameworks Driver, https://github.com/ofTheo/ofxKinect
   • Limitations of Kinect, http://www.iheartrobotics.com/2010/12/limitations-of-kinect.html
   • Color/Depth mapping, http://graphics.stanford.edu/~mdfisher/Kinect.html
   • http://sy-lab.net/#1102351/roboScan
2. 3D scanner on wiki, http://en.wikipedia.org/wiki/3D_scanner

Visualization

1. Texture mapping video to surface model
2. Display vessel beneath the surface

System Integration

Putting everything together

Action Items