Hi,

Measuring similarity or distance between two information entities is a core

requirement for all information discovery tasks (whether IR or Data mining). Use of

appropriate measures not only improves the quality of information selection but it

also help reduce the time and processing costs.

Such measures may be used, for example, for plagiarism detection.

Cosine similarity is a popular vector based similarity measure in text mining and

information retrieval. In this approach compared strings are transformed into vector

space so that the Euclidean cosine rule can be used to calculate similarity. This

approach is often paired with other approaches to limit the dimensionality of the

vector space. Cosine similarity is a measure of similarity between two non-zero vectors of an inner product space that measures the cosine of the angle between them.

For example, in information retrieval and text mining, each term is notionally assigned a different dimension and a document is characterized by a vector where the value of each dimension corresponds to the number of times that term appears in the document. Cosine similarity then gives a useful measure of how similar two documents are likely to be in terms of their subject matter.

similarity=1-distance

Pearson similarity measure is based on how much the ratings by common users for a pair of items deviate from average ratings for those items.

Adjusted cosine similarity measure is a modified form of vector-based similarity where we take into the fact that different users have different ratings schemes; in other words, some users might rate items highly in general, and others might give items lower ratings as a preference. To remove this drawback from vector-based similarity, we subtract average ratings for each user from each user's rating for the pair of items in question.

The cosine approach provides a kind-of Bayesian regularization for the metric, ensuring that the similarity is not fully decided only by the (potentially small) subset of items the two users have in common. This is not natively present in Pearson correlation (although it can be added by significance weighting), so in contexts where users tend to have very differing sets of items in their profiles, (uncorrected) Pearson would perform worse, in principle.

Pearson correlation is simply the cosine similarity when you deduct the mean. This is important because the mean represents overall volume, essentially. If overall volume is of substantive interest then you would want to use cosine similarity (or some measure that keeps overall volume). Often it isn't of substantive interest, though, and removing it is desirable.

"Adjusted cosine" similarity is done by subtracting the mean before computing the cosine formula. In that sense, adjusted cosine would have the same mathematical formula as Pearson correlation. However the results will not match if the formula is applied differently, namely if the set of used ratings is not the same.

http://www.iicm.tugraz.at/thesis/bilal_dissertation.pdf

http://www.cs.carleton.edu/cs_comps/0607/recommend/recommender/itembased.html

Hope this helps.

Dear Alexander Egoyan,

Thanks for your valuable time and sharing useful knowledge. Such information helped me understanding the trends and usage of such similarity measuring techniques. For further detail, I will read the thesis file.

Regards

Zafar Ali

Similarity measure is one of the fundamental exercise of data mining and information retrieval. An efficient similarity measure results to less time and processing efforts.

pearson and cosine are most widely used and popular similarity measures.

pearson correlation coefficient defined as the covariance between two vectors divided by their standard deviations.

cosine similarity defined as the angular similarity between two vectors. angle 0 defines match and 90 otherwise.

adjusted cosine or cosine over mean centred vectors is very much similar to pearson similarity except if two are calculated over different set of rated vectors.

person correlation works over co-rated vector where as adjusted cosine consider all the rated vector.

a useful property of cosine correlation is that a subset of a larger data set clusters in the same way it did in the big set, because the data is not centered and normalized again (which is what Pearson would do).