Hi,

Yes, categorizing the methods based on their properties/challenges helps you to choose the appropriate evaluation metrics. However, choosing the evaluation metrics depends on your proposed approach, its counterparts, and the challenge(s) you have attempted to tackle with it (them). Reading the following paper might be useful.

http://www.jmlr.org/papers/volume10/gunawardana09a/gunawardana09a.pdf

Dear Hadi Shakibian,

I agree with you, we should select the evaluation method by considering the requirements and needs of the user. However, there are enormous approaches for evaluating recommendation results, however, sometimes it become a daunting task to choose an appropriate metric. various classifications are available for evaluation metrics but, no proper agreement exist and hence there is no such benchmark to follow. Some researchers make use of surveys, while others follow classification and accuracy metrics. Like other systems, some rules need to be introduced for assessing the results of these systems.

Thanks

The most commonly used quality measures are the following

: (1) prediction evaluations

matrices are : MAE, RMSE, coverage, NMAE

(2) evaluations for recommendations as ranked

lists.

matrices are: cumulative gain, half-life

(3) evaluations for recommendation

as sets, 

Recall, Precision, Reciver Operating Charateristics, fall out

Some others are diversity and novelty.

However, there are a few standard framworks apart from Herlocker et. al which actually depicts a lack of formalizations. 

Dear Supratip Ghose,

You have provided very good classification example but, i have observed that there are many survey papers on evaluation metrics, most of these research studies have classified metrics by adopting different approach. Some researchers place one metric in one class while other place the same metric in a different class. Still researchers are not agree on a benchmark framework that could be used for evaluation. Can you please give an examples of such survey papers which you think are more acceptable in this domain.

Regards

Zafar Ali

That's why I said that there is a lack of formalizations...However, evaluations and validations all emanate sometimes from the requirements..it is flexible instead of rigid...both standardizations and formalizations are required according to many authors,  however...