[vtkusers] Registration of projected textures with vtk? (2d-3d registration)

[kc]

Hello,
I am new to vtk and am looking for information regarding some questions I have
about using vtkProjectedTexture and ICP together to do 2d-3d registration.

I have a mesh (reconstructed using marching cubes) and some images of the
original object that the mesh corresponds to.   I am projecting the image as a
texture back onto the surface of the mesh using vtkProjectedTexture.  The
problem is that in order to map the textures onto the irregular surface in an
acceptable way (without distortion etc.), the projector has to be placed
accurately.  I would like to automate the process of accurately projecting the
texture onto the surface, that is, I would like to fit the contour of the
projected texture map to the silhouette of the 3d model (appropriately posed),
and use the matrices that take the image contour to its location on the mesh to
fine tune the placement of the projector.  I guess the problem can be viewed as
one of 2d-3d registration (or pose estimation), whereby I’d like to use some
initial (guess) placement of the “texture projector”, basically the location in
world coordinates where the actual camera would be placed, along with
vtkIterativeClosestPointTransform to fit the texture map onto the surface in an
accurate way.  I have some of the camera calibration info (intrinsics but no
extrinsics).  What I need, in part, are the rotation and translation matrices
(R, T) from world to camera coordinates.

I’m thinking that in order to do this one would project the mesh (via the
guessed approximate R and T) onto the image plane, use ICP (between the outline
of the mesh and the image contour), and than transform back using the inverses
of R and T, along with the transformation that is output by
vtkIterativeClosestPointTransform.  This should yield the pose of the mesh that
fits the projected image contour.

I have seen some papers where it looks as if some have used vtk to texture
complicated looking industrial parts, so this seems like something that might
be done routinely by some on this list.  Has anyone on this list used vtk in a
similar context, and if so, is there any preexisting code or examples that I
might find helpful lurking about somewhere?  Also, if anyone who has tackled
such a problem has any pointers or suggestions as to the approach they’d be
much appreciated.  Perhaps I am overcomplicating things?

Thanks in advance!

-kc