Hi Lei,

I am not sure what is the main problem in your application, so will try to address each of the points you made in your query, hoping that what you need to know is inside these responses:

1) "The monocular slam's initial coordinate system is random and scale-unknown".

It is true that single camera does not suffice to obtain the scale of a scene geometrically reconstructed from correspondences in two (or more) views. In other words, the baseline (i.e., the vector connecting the two camera centers) has unknown scale/norm. So we usually pick a norm for this vector (usually 1 if no other information is given) and everything else fall;s into place. However, the initial reference frame is not random. It is simply taken as the identity orientation and the zero-vector [I | 0] in terms of position/translation, simply because there is no other reference to be expressed it in terms of. So this is easy. Coordinate frames have a meaning only in relation to other coordinate frames.

The usual SLAM initialization will entail the recovery of a homography (see PTAM) or essential matrix (see my technical report here: https://www.plymouth.ac.uk/uploads/production/document/path/8/8593/Relative_Pose_-_George_Terzakis.pdf ). Of course, the baseline will have arbitrary scale, which typically is chosen as 1 and the rest will fall into place.

2) "But I want to know how to transform the initial coordinate system to another coordinate system(from a marker, like chessboard)".

Suppose that you have two coordinate frames placed at positions b1 and b2, with respective orientations R1 and R2 (containing the direction unit vectors of each frame as columns), then a point M1 in frame #1 has coordinates M2 in frame #2, related with (using MATLAB syntax):

M2 =R1*R2' * (M1 - ( b2 -  b1 ) )

Now, if the first frame is taken as the global (or world) reference (hence, R1 = I3 and b1 = 0), then the transformation becomes: M2 = R2' * M2  * (M1 - b2).

In the case that you are describing, you have to use the general case, since the PnP will give a pose in terms of the local coordinate frame of your landmarks.

3) "But the mappoints reconstructed by the marker during the SLAM's initialization were untrusted.And the graphic optimization would change mappoints' position to make a smooth trajetory by minimal the projection error."

You are using the PnP to obtain camera pose (i.e. orientation and position) from known points. You mention that these points are "untrusted". I suppose you mean that their true position is uncertain (?). I suppose you mean that their *tracked projections* on the first two images in the sequence are uncertain. Yes, that is correct and moreover, it would propagate uncertainty into your camera pose estimate. This estimate can only be improved by tracking these points further in subsequent frames, repeat the PnP for these camera frames to obtain the respective camera poses and then run *bundle adjustment* to improve the camera position estimate (usually BA improves both camera pose and feature locations, but since you know your landmark locations, BA should improve only the poses and potentially the 3D locations of features that have been detected in new frames, but did not belong in your initial set of landmarks).

Feel free to ask more, or take a look at my PhD thesis (chapters 2-3) for more information.

I really hope the above is helpful.

Hi George,

Thanks for your great answer! And sorry for my late reply.I would give more detail information according your answer.

1) "The monocular slam's initial coordinate system is random and scale-unknown".

I mean that the monocular slam's coordinate system's x-y-z axis direction was not definite and scale-unknown.Because we don't have definite initial reference frame.

2) "But I want to know how to transform the initial coordinate system to another coordinate system(from a marker, like chessboard)".

In my situation,I'm studying about the ORB-SLAM2 and I want to do some AR test base on it.For example,i want to draw a cube at the origin of the SLAM's coordinate system.Since the SLAM's coordinate system's direction was not definite,so i want to use a marker(like chessboard) to make it fixed and draw a cube on the marker's origin.But i haven't done with it.Please give me some help.

3) "But the mappoints reconstructed by the marker during the SLAM's initialization were untrusted.And the graphic optimization would change mappoints' position to make a smooth trajetory by minimal the projection error."

In order to get a fixed SLAM's coordinate system,I've modified the initialization of the ORB-SLAM2 by triangulating the initial 3d-mappoints with camera pose calculated from marker(by PnP).Then,i think the slam's coordinate system's direction was definite and scale-known.So i drew the origin's x-y-z axis,but the result shows that it has a obvious offset on origin's position and x-y-z's direction.

During the initialization,the SLAM using graph optimization(BA) to optimize the 3d-mappoints and this would change the 3d-mappoints' position.So i said the 3d-mappoints were untrusted.

Thank you!

Hi Lei,

No problem at all. Here's a few comments on your updates:

* As I noted in my original response, you should not be worried about the fact that the coordinate frame is "unknown". A coordinate frame is is always expressed relative to another coordinate frame and in that sense, the frame of your markers is also "unknown", since, in turn, there is no other frame to express it in terms to . Thus, we always take the first camera pose (or the frame of your markers) in the sequence as the identity and then everything else is expressed in terms of that. So, using the PnP does not help you in this regard. However, it will provide scale for your initial scene, which may or may not be important, depending on your application. Just to re-iterate my previous comments, scale is essentially determined by the norm of the baseline vector between the first two camera frames, of distances between the markers. This scale factor will propagate linearly throughout the rest of the SLAM sequence onto the reconstructed points and camera positions. So, you can initialize the SLAM with standard 2-view geometry (essential matrix - homography, etc) and the use a baseline nom 1 to obtain your scale.

What is  a bit of a problem, is that 3 points are very - very few to do camera camera pose tracking (actually, 3 is the minimum  PnP configuration). To give you an idea, for efficient visual SLAM, a minimum of 100 points would be more reasonable. Of course, ORBSLAM will detect new points as it goes, but the initial pose estimate will be probably very weak. And this means that subsequent bundle adjustment is going to be very ill-posed.

The above bring me to your next point:

*  There is no need to use the PnP to initialize your visual SLAM to obtain a coordinate system, especially if you want to add AR. As I said, there is no such thing as "unknown coordinate system". In fact, I am very sure ORBSLAM will express everything in terms of the first camera pose (i.e., the coordinate frame of the pose of the camera at the beginning of  SLAM). Thus, you will be expressing your AR models in terms of  this coordinate frame.

* Now, about scale: Again, the scale factor is simply a multiplicative factor that will "shrink" or "grow" the reconstruction and camera position. However, it will not change relative distance ratios and therefore you can always place your AR models without knowing the actual; scale of the scene (we basically choose a scale that appears realistic, such as 1, or 0.5). Thus, you can use default 2-view visual SLAM initialization provided by ORBSLAM and move on.

* If you want to see an alternative in action, why not try PTAM? I have actually modified the original PTAM into a new project which uses only OpenCV. The code has been "retrofitted" line-by-line and works and compiles without any dependencies other than OpenCV. It uses the original PTAM models, but you can easily change that. Of course, it works only under linuc, which is the case with ORBSLAM. Here's the link in case you want to use it:

https://github.com/terzakig/gptam

Hope the above help

All the best with your project.

Hi George,

Thanks for your answer!I've learned some about visual SLAM from your answer.

*I realized that the PnP from a marker doesn't give any help except getting scale of the world coordinate and the SLAM's poses were all related to the first frame.But as i know,the scale was determined by the camera's movement distance between the first two camera frames.

*The string "unknown coordinate system" i said was mean that the SLAM's coordinate system was not fixed during the initialization in every times.For example,i initialized the SLAM successful at the first time in the scene1 and the SLAM's coordinate system marked as 'C1'.Then i shutdown the application and intialized the SLAM at the second time in the scene1 again,the SLAM's coordinate system marked as 'C2'.The 'C1' and 'C2' was not same because of uncertain first frame.

In my project,i want to put the cube on a marker(like chessboard,certain position instead of the origin of the SLAM's coordinate system)  in every times.In other words,i want to get the transformation between the SLAM's coordinate system and the marker's coordinate system.But i haven no idea to do that.

Did i understood it wrong?Please tell me.

*About scale,i'm studying about monocular SLAM and it is difficult to get the accuracy scale of the world.

*I've cloned your project from the github.I modified the cmakelists file about include path,lib path and c++11(Btw,in some source file,the opencv include file path may incomplete in terms of the cmakelists file.For example,in 'OpenCV.h' the header files may be '' instead of ''). I built it done and it runs good.I would take more time to study with your code.

Thank you!

Hi Lei,

I am very happy to have helped!

About your first point: Correctly. If you want - for the sake of the application at hand- to have a reconstruction relative to the cube, then yes, you need to initialize by triangulating known landmarks. However, they would have to be very strong features such as chess-board corners in order to be able to track them with high accuracy. Also, there would be ambiguities depending on how the corners are arranged on the cube.

About scale. I wouldn't worry too much about it. There are many ways to add scale without markers. For instance, you can use a kinect to gain depth measurements, or employ a stereo rig.

About my version of PTAM. Hope it will be of help to you! Bear in mind that PTAM initializes with ambiguous scale (actually initial baseline vector is 0.5 if I remember correctly).

All the best with your project!

George

Hi George,

I'm sorry for my ambiguous description about my first point.

As in your version of PTAM,you draw four eyes on the dominant plane(detected from the 3d point cloud).Is there any method to put the four eyes on a certain place(a marker,like chessboard) instead of the dominant plane(a transformation between them)?If i've got the transformation,the marker could be took away,and the four eyes still on the original position(alignment by the marker).

Thank you!

Lei Zhu

To address this problem you need to make a few modifications, but not many thankfully! You have to modify the code to recognize the markers and their arrangement in 3D space. The latter is a bit tricky, but not hard.

How would I do it:

I would add a special routine to check if a feature is a marker inside the tracker (tracker.cpp). Then, if this feature becomes a mappoint (there are several of checks inside the tracker and the mapmaker that may reject this point), it will be tagged as a "landmark" in the map.

Then, in the mapmaker (mapmaker.cpp), Instead of doing the SVD on the point cloud, you can go through the map-points and pick only the ones that are marked as landmarks to obtain your plane. The rest is already there!

And the great thing is that you don't need to use the PnP to obtain scale for your models! It will come for free when you have found your landmarks in the map!

I have added many comments in the code and that could help put yours in.

Hi George,

Thanks for your answer.With your idea that means reconstructed the keypoints on the marker,and detect a dominant plane from the landmarks that on the marker.Finally,i could put the four eyes on the marker.

But in this way,the accuracy of the landmarks would have some influence on the plane.I mean that the plane might not parallel to the marker and have a offset.

I would try with your idea.When i done with that,i would give you some more information.

Finally,thanks for your great help!

Best wishes,

Lei Zhu

Yes, would be very interested to hear your progress on this problem!

Best of luck!

Hi George,

I've done with your idea.As expected,the dominant plane detected from the point cloud on the marker region have a slant with the marker.Now i'm trying to get the rotation between the SLAM's coordinate system and marker's coordinate system without translation.And i think it's not difficult.

Sounds great! I am happy to have helped!

I am currently facing the same problem, my approach is to save a matrix when the initialization is successful, which can transform marker coordinates to slam coordinates and then use Tcw which is returned by orbslam2 to transform it into the current camera coordinates. But the virtual object is not static it is moving with my camera movement , whether it is because the scale is not consistent or not. Hope to get help, thanks