Couldn't you basically use the same approach, i.e. comparison to a golden standard. If so, you basically have to identify the possible measures, similar to precision and recall in the case of classification. Most obviously, I could think of:

- having ranked something at the right position;

- having identified the correct order between two items;

- having identified the correct 'distance' between two items; and

- trying to extend the above towards more elements.

The easiest thing I could think of is a rank correlation coefficient. Or, if you have relevant and non-relevant items on the list, you can use average precision or AUC.

Thanks a lot for all these suggestions... and papers !

@Felix Weninger : could you be more explicit ? How are defined "rank correlation coefficient", "average precision" and "AUC" ?

Regards,

Matthias.

(Spearman's) rank correlation = Pearson correlation of the ranks (highest score = 1, ...)

average precision = area under the precision-recall curve when considering the 1, 2, ..., highest ranked results

AUC = similar, but integrating the TPR/FPR curve.

For once, I'd recommend reading the Wikipedia articles on binary classification and information retrieval, they are pretty well written imo.

You might consider the Kendall rank tau correlation test which gives you statistical test (including significance value) whether or not two rankings are the same or differ significantly. This has the advantage that in the end you can state your ranking *not* to diverge significantly form the gold standard, which might be more difficult to judge with AUC or a correlation coefficient.

A brief definition and example is provided in http://www.utdallas.edu/~herve/Abdi-KendallCorrelation2007-pretty.pdf -- you might want to have a look at this. I like that short introduction and crisp definition.

You can simply consider the relation between any two pair of items, if corrrectly detected put a 1 else 0. Sum them up and divide by number of pair of items