Align a 3d mesh model with its image

Code : [github/mesh_image_align]

Today I am trying out this one simple idea I have in my mind. Given an image of an object (I will use banana) and it’s 3d mesh model the objective is to get the pose of the object relative to the camera. Once you have it, you can project the 3d points with this estimated pose to check the alignment.

I downloaded a 3d model of banana from : here. I took a picture of the banana myself with a camera whose intrinsic parameter I know. Next manually marked some 3d points on the image. Thus, I know the 3d-2d alignment. Recently, semantic alignment has been a research topic. But I bypass all that and just manually mark the 3d-2d correspondences. In the future, when there are good methods to do it, can substitute for the manual marking.

Next, I used a PnP (Perspective-n-point) to align the mesh with the image, ie. to compute the pose between the 3d points and the camera. Here is the result, ie. projection of a mesh ( $latex ^{w}X ) model using the estimated pose:

$[u;v] = \pi ( ^{c}T_w \times ^{w}X ) )$