By comparison against a set of examples. Store 3 comparable

items, say A,B,C, along with the information whether B or C

should be judged to be closer to A. Assemble a set of such

evaluation examples. Let your algorithms determine the

similarity between every A and B, A and C, and add a +1 to its

score if it is the desired result, otherwise a -1.

Regards,

Joachim

I would agree with Joachim's answer (i.e. show that your algorithm better reflects some independent, possibly subjective, measure of similarity than the alternatives), but add some sort of benchmark for computing the measure in sets of N items. Path similarity metrics can be complex, and it's no good having a very intuitive metric if we can't actually deploy it in real-world data.

One other option that I've seen, but can't quite flesh out with the details in your question, is to show that your algorithm drives better performance in some specific application than the alternatives - that your metric makes it easier to solve some other problem.... Not a terribly specific suggestion, I know, but worth mentioning?

Usually a similaritry metric is used to do something else, like clustering, classification, ranking, etc. Now you have to find a nice problem involving e.g. clustering. You choose an accepted clustering method and an appropriate evaluation method for the cluster quality. Then you repeat the experiment several times, keeping everything constant, changing only the similarity measure between the items. If the new measure gives the best results, it was the most appropriate measure for the given data set and the given task. Ideally you should repeat the experiment for several tasks and several data sets.

For many tasks there exist common benchmark data sets. It would be best to use such a data set. E.g. for word similarity you should include the TOEFL data (http://aclweb.org/aclwiki/index.php?title=TOEFL_Synonym_Questions_%28State_of_the_art%29).

We have done some small experiments following this pattern to compare different similarity measures:

* https://www.researchgate.net/publication/236625601_Distributional_Similarity_of_Words_with_Different_Frequencies?ev=prf_pub

* https://www.researchgate.net/publication/221466114_Instanced-Based_Mapping_between_Thesauri_and_Folksonomies?ev=prf_pub

* https://www.researchgate.net/publication/4374762_Topic_Detection_by_Clustering_Keywords?ev=prf_pub

Conference Paper Distributional Similarity of Words with Different Frequencies

Conference Paper Instanced-Based Mapping between Thesauri and Folksonomies

Conference Paper Topic Detection by Clustering Keywords