I need to evaluate a pure content-based recommender system for documents extraction (It may also be seen as a search engine) that gets top N results based on a similarity score. I know there are some metrics like HR@k, accuracy@k, NDCG@k, CTR, etc. However, if I understand right, all those metrics require a pre-evaluation from expert coders, rating score for documents (e.g., scale from 1 to 5) or click pattern from users.
This content-based recommender systems have no user (yet) to rate/click on query results, and I can not understand how the expert coders can provide ratings for every document against every possible query.
Are there any means to evaluate such types of content-based recommender systems?
Dear Chanaa Abdessamad
I suggest you to go through these articles first:
1. https://towardsdatascience.com/recommendation-systems-models-and-evaluation-84944a84fb8e
2. https://towardsdatascience.com/techniques-for-content-based-recommender-systems-64f812d2b5a0
3. https://www.sciencedirect.com/science/article/pii/S1110866515000341
4. https://stackoverflow.com/questions/6167855/how-to-evaluate-a-content-based-recommender-system
5. https://medium.com/sciforce/inside-recommendations-how-a-recommender-system-recommends-9afc0458bd8f
6. https://medium.com/@jwu2/content-based-recommender-systems-an-overview-84a65eaeebb9
Kind Regards
Qamar Ul Islam
The top-N documents with the highest similarity scores is the recommended list for a user based on a specific query. After that, you can identify a value for each document in the recommendation list either relevant or non-relevant based on the similarity with the content of the query and the recommended document. You can select experts that have experience on the field that you work. After they assign a value for each document, you can easily calculate the accuracy performances: Precision, Recall, and F-measure, for your CB recommender system.
Good Luck !
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs