[vtkusers] Real-Time Image Warping

[David Gobbi]

Hi Jorge,

When using a reduction factor with the grid, you should be careful
about using anything but grid->SetInterpolationModeToCubic().  Linear
interpolation will result in a transform that isn't smooth.

I'm not sure why you see a performance decrease when you use a larger
reduction factor, but a factor of 4 is what I use, too.  Also, the
performance is about what I would expect.

The VTKEdge code doesn't do image warping, or at least, not as far as
I know.  I'm guessing that you would have to write a custom vertex
shader for VTK, which is something that I've never tried.  If you have
any success, though, I'd be glad to hear about it.

   David


On Mon, Apr 12, 2010 at 3:02 AM, Jorge Ballesteros (Motiva)
<jballesteros at motivando.me> wrote:
> Hi David,
>
> First of all, thank you very uch for your advice. I have reduced the
> resolution of the transform grid and the performance has improved a bit
> (about 15 Hz) . Even though i have to set the reduction factor to 4.
> Whenever i try to use a grater factor, let's say 8x or 16x, the performance
> decreases. I got the best results with the following parameters:
>
> * Image resample from 512x512x100 to 256x256x100 setting the output extent
> to 256x256x5.
> * Reduction factor = 4. Grid spacing 4x and extent reduced by same factor.
> * InterpolationModeToLinear
> * GridScalarTypeToFloat
>
> I think i will try to use GPU-based calculation, with VTKEdge package. I
> have already used VTKEdge for Volume Rendering and the results are great.
> Any hint on how to do image warping with VTKEdge?
>
> Thanks for all,
>
> Jorge
>
> David Gobbi escribió:
>>
>> Hi Jorge,
>>
>> I notice that in vtkTransformToGrid, you are creating a grid that has
>> the same resolution as the input.  That defeats the purpose of using
>> vtkTransformToGrid, since it means that you are evaluating the thin
>> plate spline for every voxel in the image.  You should set the
>> vtkTransformToGrid spacing to 8x or even 16x the image spacing, and
>> reduce its extent by the same factor.  Then, for vtkGridTransform use
>> SetInterpolationModeToCubic().
>>
>> Also, you might see some small improvements with
>> SetGridScalarTypeToFloat().  That will speed up the creation and
>> evaluation of the grid transform.
>>
>>   David
>>
>>
>> On Fri, Apr 9, 2010 at 4:19 AM, Jorge Ballesteros
>> <jballesteros at motivando.me> wrote:
>>
>>>
>>> Hi David,
>>>
>>> As you indicated I have rescaled the image to 256x256x98 and the
>>> performance
>>> is better. For an 256x256x3 input image the warping frame rate is about
>>> 25Hz.
>>>
>>> Even though is a big improve, i would like to keep on improving the
>>> performance, but not downsampling the image. I am using
>>> vtkThinPlateTransform as the the input to a vtkTransformToGrid and then
>>> using a vtkGridTransform as the reslice transform to the vtkImageReslice
>>> filter. I have been trying several pipelines but this is the one that
>>> returns the best results. Not sure if these are the proper functions to
>>> perform a 3D image warp. Am i right? Below is an extract of the code:
>>>
>>> vtkSmartPointer< vtkDICOMImageReader> reader =
>>> vtkSmartPointer<vtkDICOMImageReader>::New();
>>>  reader->SetDirectoryName(argv[1]);
>>>  reader->Update();
>>>
>>> vtkImageData * input = reader->GetOutput(); //256x256x3
>>>
>>> Downsampling...
>>>
>>> double point[3];
>>> double target[3];
>>> ...
>>>
>>> int limit = 15;
>>>
>>>  vtkSmartPointer< vtkPoints > p1 = vtkSmartPointer< vtkPoints >::New();
>>>  p1->InsertNextPoint(point[0]-limit, point[1], point[2]);
>>>  p1->InsertNextPoint(point[0], point[1]+limit, point[2]);
>>>  p1->InsertNextPoint(point[0]+limit, point[1], point[2]);
>>>  p1->InsertNextPoint(point[0], point[1]+limit, point[2]);
>>>  p1->InsertNextPoint(point[0]-limit, point[1]-limit, point[2]);
>>>  p1->InsertNextPoint(point[0]-limit, point[1]+limit, point[2]);
>>>  p1->InsertNextPoint(point[0]+limit, point[1]-limit, point[2]);
>>>  p1->InsertNextPoint(point[0]+limit, point[1]+limit, point[2]);
>>>  p1->InsertNextPoint(point[0], point[1], point[2]);
>>>
>>>  vtkSmartPointer< vtkPoints > p2 = vtkSmartPointer< vtkPoints >::New();
>>>  p2->InsertNextPoint(point[0]-limit, point[1], point[2]);
>>>  p2->InsertNextPoint(point[0], point[1]+limit, point[2]);
>>>  p2->InsertNextPoint(point[0]+limit, point[1], point[2]);
>>>  p2->InsertNextPoint(point[0], point[1]+limit, point[2]);
>>>  p2->InsertNextPoint(point[0]-limit, point[1]-limit, point[2]);
>>>  p2->InsertNextPoint(point[0]-limit, point[1]+limit, point[2]);
>>>  p2->InsertNextPoint(point[0]+limit, point[1]-limit, point[2]);
>>>  p2->InsertNextPoint(point[0]+limit, point[1]+limit, point[2]);
>>>  p2->InsertNextPoint(target[0], target[1], target[2]);
>>>
>>>
>>>  vtkSmartPointer< vtkThinPlateSplineTransform> transform =
>>> vtkSmartPointer<
>>> vtkThinPlateSplineTransform>::New();
>>>  transform->SetSourceLandmarks(p1);
>>>  transform->SetTargetLandmarks(p2);
>>>  transform->SetBasisToR();
>>>
>>>  //Convert the thin plate spline to a grid
>>>  vtkSmartPointer<vtkTransformToGrid> transformToGrid =
>>> vtkSmartPointer<vtkTransformToGrid>::New();
>>>  transformToGrid->SetInput(transform);
>>>  transformToGrid->SetGridExtent(-2*limit, 2*limit, -2*limit, 2*limit, 0,
>>> 0);
>>>  transformToGrid->SetGridOrigin(point);
>>>  transformToGrid->SetGridSpacing(input->GetSpacing());
>>>  transformToGrid->SetGridScalarTypeToShort();
>>>
>>>  vtkSmartPointer<vtkGridTransform> gridTransform =
>>> vtkSmartPointer<vtkGridTransform>::New();
>>>  gridTransform->SetDisplacementGrid(transformToGrid->GetOutput());
>>>
>>>
>>> gridTransform->SetDisplacementShift(transformToGrid->GetDisplacementShift());
>>>
>>>
>>> gridTransform->SetDisplacementScale(transformToGrid->GetDisplacementScale());
>>>
>>>  vtkSmartPointer< vtkImageReslice > reslice =
>>> vtkSmartPointer<vtkImageReslice >::New();
>>>  reslice->SetInput(input);
>>>  reslice->WrapOn();
>>>  reslice->SetResliceTransform(gridTransform);
>>>  reslice->SetInterpolationModeToLinear();
>>>  reslice->SetOutputSpacing(input->GetSpacing());
>>>  reslice->SetOutputExtent(0, 255, 0, 255, 0, 2);
>>>  reslice->Update();
>>>
>>> Thanks for your support,
>>>
>>> Jorge.
>>>
>>>
>>> David Gobbi wrote:
>>>
>>>>
>>>> Hi Jorge,
>>>>
>>>> For real-time image warping in VTK, you would probably have to go with
>>>> 256x256 images in order to achieve the 10Hz goal that you are aiming
>>>> for.  For warping a 512x512x100 volume in real time, I think your only
>>>> option would be to use the GPU.
>>>>
>>>>  David
>>>>
>>>>
>>>> On Thu, Apr 8, 2010 at 4:38 AM, Jorge Ballesteros
>>>> <jballesteros at motivando.me> wrote:
>>>>
>>>>
>>>>>
>>>>> Hello all,
>>>>>
>>>>> I have been trying to perform a real-time image warping with no
>>>>> success.
>>>>>  I
>>>>> have managed to warp a 512x512x100 Image but the computation time it
>>>>> takes
>>>>> is very high. Then I extracted a portion of the image, just 3 slices
>>>>> and
>>>>> warped the image but still don't meet real-time requirements.
>>>>>
>>>>> FYI i have been following VTK Examples,
>>>>> Hybrid/Testing/TestGridWarp3d.tcl,
>>>>> using the vtkImageReslice class with a vtkGridTransform as the input
>>>>> transform and a 512x512x3 input image, obtaining a ~10 Hz frame rate.
>>>>>
>>>>> Don't know if there is any other way on doing this. It would be really
>>>>> helpful if you could point me on how to achieve it.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Jorge
>>>>>
>
>