Can you capture an image of a '3D" calibration chart (2 planes with checkerboards glued to each other orthogonally). then, you can extract the correspondences between the pixel locations of checkerboard intersections and their physical (x,y,z) position on the 3D chart and get the camera pose matrix.

I could calibrate the camera. My problem is different. given the calibrated camera, I need to estimate the 6DOF of a given CAD object appearing in the scene from the 2D image.

One way to do this could work somehow by extracting and matching 2D edges to their hypothesized position on the CAD and rejecting bad hypotheses... I know from the literature that this approach could work but:

Regards,

Bruno

this could be useful (they extract the 3D model first and the camera pose from which this was captured), the 2D pose of the new object image is matched with the 3D mesh using descriptors and RANSAC

http://docs.opencv.org/3.1.0/dc/d2c/tutorial_real_time_pose.html

I had already try this approach for another project needing tracking.  The problem is that it relies on natural points detection matching (from feature) between the 2D image and the object to match.  It did not work that great and also relied on the assumption that the tracked object was a rectangular prism.

Our CAD object has no texture associated to it and will have curved surfaces (pipes). That's why we would need to match silhouette edges and real edges from the CAD to the image.

For people interested, I uploaded the original Goad paper.  More useful for the idea than the technical information it contains (implemented on the PDP-10).  Should be amazingly fast on today's computers!

https://en.wikipedia.org/wiki/PDP-10

You can project light patterns on your pipes using a light projector, to create textures which can be matched 

CAD model of manmade products can be B-rep  by STEP standard, it can be brepresented surface AAG, now surface is closed with edges, its project aspect can be matched with the real image (edges).

M. Ramachandra,

Projecting light patterns will not be physically possible in our application. Also, many structures will be made from metal reflecting light differently from varying an angle making stereo systems based on color or white light patterns unsuitable.

M. Wang,

When you say: "its project aspect can be matched with the real image (edges)."

It's exactly the problem I have.  I already have the exact CAD model of the structure appearing in the image I have.  My problem is the matching step between the CAD (no texture or color) and the appearance the real object has on the picture.

Sadly, your suggestion is like having my problem already solved...

Thanks Max!  That's a very awesome answer!

I also had other answers from Alberto Pretto about code implementing his paper "D2CO: Fast and Robust Registration of 3D Textureless Objects using the Directional Chamfer Distance".  Mister Pretto said that his group will release the refactored version of the code under BSD license!

There is also a paper named "Fast 6D pose estimation for texture-less objects

from a single RGB image" which works on the same kind of problem.

Again, thanks for having taken so much time to explain the work you did on the problem.  I hope you will find the following links interesting!

Bruno

https://bitbucket.org/alberto_pretto/d2co.git

Conference Paper Fast 6D pose estimation for texture-less objects from a sing...

Conference Paper D $$^{2}$$ CO: Fast and Robust Registration of 3D Textureles...

May I ask how you solved the problem? Can you recommend an approach (we are currently working on a similar problem)? I hope to hear from you.

Thanks!

Hello Axel,

We finally used the 3D reconstruction from the Microsoft Hololense and develop a 3D manipulator to position the CAD model close to the actual physical object and rely on ICP to refine the match.

Since then, I know that Dr. Alberto Pretto has continued to improve and generalize his library which does exactly what I intended to do at the start of the project. If I were to do it again, I would probably start from there:

https://bitbucket.org/alberto_pretto/d2co/overview

Hope that helps!

Hi Bruno. I tried to build https://bitbucket.org/alberto_pretto/d2co/overview Alberto Pretto code but got some errors. Did you try it?

Hello Bojan Andonovski ,

I did not try the code. As I said, we did something else back then but I see that the Dev branch has some development in 2020 so there is still at least some development/maintenance going on.

I think the best for you would be to try contacting M. Alberto Pretto in person and see if he can help you...

https://www.dis.uniroma1.it/~pretto/

Good luck!

is this code can solve the problem when the objects are one on top of the other? So when objects are to identical and one on top of others but with different ent orientation , and I haveto get the orientation and the position of the on on the top. Would wll be working in that case?

Again, as I'm not the author and haven't try it, you should:

I cannot tell you more than that!

Good luck!

ok thanks. Tried to contact the authors. Hope they will answer back

Bojan Andonovski , it would be great if you could update this discussion with your results and observation about this method to help future researchers!

Good luck!

sure

I tried to contact him but no answer, Was not able to compile his code. So, at this stage, I'm using the Deep Learning and PCL library methods to get the 6D Pose. The results I get are satisfying and serve the purpose as I know roughly the coordinates of the objects to be picked and placed. In the future might explore the approach with the 3D CAD model and a 2D single image.

Hello Bojan Andonovski ,

Have you also tried to contact Pretto's associate, Marco Imperoli :

[email protected]

Also, Alberto Pretto is now consulting for IT+Robotics Srl and has another e-mail:

[email protected]

Maybe you would have more chances with this address?

ok. thanks. Send email to that new address. See if they respond

Yes, it is possible to reconstruct the 6-DoF pose from a single image if you know the 3D CAD and camera position.

The image comparison algorithms, such as SSIM (Structural Similarity Index), can be your essential helper to sort out the closeness of the 2D photo and a 2D projection from a 3D pose. There are two ways to do it, I think.

The searching methods for both are the same. It can be any continuous optimization algorithm. Examples include the Genetic Algorithm, Particle Swarm Optimization, etc., which are easy to implement. However, I recommend advanced algorithms such as CMAES (code here: https://github.com/CMA-ES/pycma , https://github.com/beniz/libcmaes) and DIRECT (code here: https://github.com/rlnx/DiRect)

Nevertheless, 3D scanning is very convenient nowadays, e.g., on your iPhone/iPad/Android and LiDAR scanners. Why not consider matching 3D CAD to 3D point clouds? Xue et al. (2019) describe a general way for that based on RMSE.

References

Xue, Lu, & Chen. (2018).

Article Automatic Generation of Semantically Rich As‐Built Building ...

Xue, Lu, Chen, Zetkulic. (2019).

Article From Semantic Segmentation to Semantic Registration: Derivat...