I want to evaluate a matching algorithm in the house dataset (http://vasc.ri.cmu.edu/idb/html/motion/house/). However, I think this dataset doesn't contains any kind of label. How can I evaluate the matching algorithm in this dataset?
The focus of a matching algorithm is to identify the same set of objects or similar objects in different images, therefore all you need to do is to provide an example to the matching algorithm, then let it go through carefully chosen images from your dataset. You can then evaluate your result using a Confusion table.
Dear Eliezer,
When you are doing template matching , you don't need to have labeled data. Because you are comparing an image with a template. The evaluation of the matching can be measured by finding Correlation (to be more accurate cross correlation) between the image and the template or Sum of Squared Error (SSE) .
If you program with Matlab , you can use their template matching functions.
Hope it helps.
Thank you for the answers. Mr. Saman Sarraf, I'm not sure if I am explaining my problem correctly. In my program I identify some interest points in the first image. My question is, how can I to know how many of these points I identify in the second image (template)? Furthemore, the template is the first image rotated...
Dear eliezer,
Your problem is, if I'm not taking it wrong, bipartite graph matching. Put boldface, matching a series of points which you tagged in your image to the points in an alternative set (which acts as ground truth). This reduces to findind the mimimum cost of the matching (that is, the list of pairs so that at least one of the sets is completely matched to the other, and the sum of their distances is minimum.
For a limited number of points (dozens), you can apply the Munkres/Hungarian algorithm, which will give you a deterministic, optimal solution. Otherwise, you can find pseudo-optimal algorithms in Martin (Phd, UCBerkeley, 2004) or Estrada-Jepson (CVPR05, IJCV09).
In any case, that information needs to be further adapted to your problem. Are the points located well? Are they too far away from their real location? This, however, depends on your application, the first step is probably the one I mentioned before: matching the point cloud in your candidate solution to those in the ground truth.
cheers,
c
Dear C. Lopez-Molina,
Actually I'm trying to modify the Factorized Graph Matching algorithm. However, I need to evaluate the impact of my modification. The demo provided by the authors use the house dataset...
Dear Eliezer,
FGM is in my to-do list. In my opinion, you really have to work on the semantics of the problem and the implications they have in the quality evaluation process. Questions need to be answered: Will there be an equal number of points? Are all of the points (at each side) matcheable to each other? Is it reasonable to have long-distance matchings? Are you penalizing on the number of non-matchings (FPs or FNs), or also on their distance?
From my experience, analyzing those questions will lead to (or greatly restrict the scope for searching) a solution. Do not hesitate in contacting for more info!
Dear Mr. C. Lopez-Molina, thank you for the answer...
Well, I don't have the same number of points for the two images, but most of the points are matchable (I did the ground-truth for myself). Furthemore, I'm thinking about using the distance between the assignment and the actual match as a metric to evaluate my framework.
dear eliezer,
first, if the number of points is different, there are several questions to be faced before any further processing. My recommendation, if interested in it, is twofold:
a) Define your task, as well as the conditions of the evaluation measure, as well as possible. Can a point be matched to different ones? How should displacements be penalized. Is it better a FP or a FN? Non trivial, although they might look like so. Then, proceed for analyzing the options in the literature, and see which matches your semantics.s
b) Take a look at metrics for binary sets (works by Baddeley, for starters).
cheers,
carlos
- Badges
- Science topic