Dear all,
We developed a context-aware content-based recommender system which computes recommendations by calculating the similarity between user profile and items description.
Given that the recommendations are built according to their descending cosine similarity scores (the higher the similarity, the better the recommendations), we evaluated the effectiveness of the recommender by adopting typical classification measures, as precision and recall.
In order to extend the evaluation of our system and to compare it with newer and better approaches, we would like to "shift" towards a recommendation model which predicts ratings, in order to evaluate it through more widespread measures such as RMSE and MAE.
How would you deal with this? How can we shift from a similarity score (e.g. 0,85) to a rating predicition (e.g. 4,21)?
Thank you all!
Dear Primal,
Actually, my question was quite different. My (content-based) recommender suggests to a target user the items with the highest cosine similarity according to the vector-space representation of user profile. As a consequence, I can evaluate the effectiveness of the system by exploiting classical information retrieval measures (precision, recall, etc.)
Now I want to "transform" my similarity scores in a real rating prediction (e.g. 0.68 similarity becomes 4 out of 5 as rating prediction). How could I do?
Dear Joao,
Thanks for your question. I already thinked about it, but I was wondering whether it was too 'naive'. Don't you think so? :-)
I assume you already have the ratings matrix (user,item)->rating.
The easiest way to compare is to do what has been said above - your similarity * 5, and see what you get. Then you can start blending in collaborative filtering into the mix, if your data is not too sparse.
Also pick your error measure carefully, depending on the problem you want to solve. E.g. what kind (in this case magnitude and sign) of errors are really important to you and are they equally important.
I think that shifting from recommendation to prediction may not be that trivial: generating predictions implies predicting how much a user will like/find useful a given item, while content-based recommendation is more of an information filtering affair.
By multiplying the cosine similarity and calling it a prediction you are assuming that the percieved quality of an item is solely based upon its similarity with other items. There is plenty of examples in which such an assumption won't hold: consider for instance Horror movies, most of them have the same plot, the same situations and often the very same actors and directors, however despite the high similarity you can still find good horror movies and crappy horror movies.
On the other hand, there exists other domains in which the potential interest can be represented by the sole similarity, for instance the domain of Scientific literature, Legal papers, Patents, and other technical documentation (when writing a paper you may cite papers that you consider crap, or a lawyer may find useful laws he/she just doesn't like), and therefore in which such shifting operation may make much more sense.
So, in my humble opinion, everything boils down at asking yourself what are you recommending: if your system recommends papers, well, the "naive" multiplication solution may just work fine, otherwise consider introducing collaborative filtering elements to predict ratings.
Hi there!
I assume that you have no rating data in your internal domain, in this case you calculating prediction wouldn't be an easy task. The output of your algorithm may not follow the nature of ratings distribution. Several algorithm may have weird output distribution, especially for content-based filtering or context-aware methods.
An idea for approximation which may be better than formula rating=1+sim*4: Shifting predictions from [0,1] to [1,5] can be done by a naive non-linear and distribution independent transformation.
It may be not the best solution but good enough to cope with any distributions.
Trick: You also can calculate the distribution of average user and item ratings (with some threshold) for both domain (your internal domain with similarities and external domain with ratings.) and applying the introduced method.
Bests,
David
- Badges
- Science topic
- Similar topics
- Correction