Visual SLAM algorithms although gained much interest last 15 years, still there seems to be a lack of good metrics that encapsulate all the performance such as dependence on robot speed? Do you know of any good metric?
This is a good question. Usually, a SLAM performance will be measured by average localization error for each point, after the full dataset has been mapped. Arbitrary performances such as the robot speed you mentioned, or computation speed, are sometimes included in literature, but they are usually specific for the input hardware and how this is processed. This is my experience with the field, but I'll stay tuned for other's views.
Thanks Raf, My point is you can have small errors by making short travel path, and increasing robot speed. In such case the metric is not suitable to compare for lower speeds and longer routes. There is a need for a robust metric.
It is very important that SLAM performances have to be measured not only for localization but also for mapping.
As mentioned by Raf, the usual way to estimate pose quality is the comparison with a ground truth. For each point, you can extract some error associated with standard deviation of the error (if your SLAM takes into account uncertainties).
Considering the mapping, again it depends on the primitive used for the map. For a simple 3D points landmark based map, the reprojection error of the 3D points in the images is the most used in case you don't have any 3D ground truth model. For processing this measure, you need to (1) estimate the camera pose for each image, (2) reproject 3D points corresponding to 2D detection and then measure the pixel error.
In case you have 3D model, you have to use distance metric between your detection and the nearest 3D object of the reference (distance point to plane, point to point, line to line..) depending the type of the map.
Considering the robot speed, it will directly impact the quality measure of both localisation and mapping. For sure, with a same algorithm, SLAM at low speed will give better result than at high speed. In my opinion to really compare two different SLAM approaches I would do the experiment at the same speed or artificially reduce sensor data frequency to make the two approaches equivalent.
For sure it exists other ways to evaluate SLAM algorithm I don't know yet. I hope that my answer will help you.
Thanks Vivet for your answer.
I have one comment, the effect of speed is opposite to your statement, low speed gives higher errors because there are only little cues available to estimate the motion.
- Badges
- Science topic
- Similar topics
- Engineering
- Control Theory