I don't think there is ONE per se correct cut-off value. The distance between your vectors depends on the vector space and therefor on the features you use to calculate the vectors. Given the definition you mentioned (0= no similarity, 1=identical), a similarity above 0.5 might be a good starting point. From here you should evaluate the similarity you calculate against your expectations (if e.g. you know what instances should be similar). Then you can see how good your feature set is and if those features, and hence the vectors they result in, really reproduce the similarity you expect.

Finding a correct cut-off value always depends on the situation and is an evaluation task you have to adapt to your data.

As Bastian suggests, there is no single cut-off value. Different vector spaces behave differently. So the best way is to have some sample data with which you can set a threshold based on your space and your task. Alternatively, based on sampling a lot of queries, you could make a determination of what is not at all relevant (e.g., the cosine is equivalent to a query made up of a random set of words) vs. potentially relevant. Cosines also do not behave linearly in vector spaces in terms of relevance. So the perceived difference in relevance to a query between vectors with cosines of .3 and .4 may be much smaller than vectors with cosines between .7 and .8 or .above.

When working on a corpus of documents, we use as threshold a formula such as :

min=avg(sim)+α⋅σ(sim)

Where α is a parameter and σ is the standard deviation. When tuned, we choose α=0.75. But, as said, it depends on the vector space.

In fact, this threshold was used as a constraint in a Grouping Genetic Algorithm to cluster these documents.

Nikola, I agree with Bastian, Peter, and Pascal. As Peter pointed out, "the best way is to have some sample data with which you can set a threshold based on your space and your task". The reason is that cosine similarities depend very much on your domain structure. Here is how you can do this: STEP 1: Randomly identify a small sample of documents. 50 may be enough, 100 is better, and so forth. STEP 2: Compute cosine similarities (CSS) between your query document (or documents, if you have more than one) and all 100 documents in your sample. STEP 3: For each of the 100 comparisons, use human assistants to obtain subjective relevance. The subjects can be students who do this for extra credit, friends who are helping you out, a "crowd" of m-Turkers if you decide to crowdsource it, or, in the worst case, just yourself. Let's suppose you use 5 friends. This will give you a data set of 100*5 = 500 rows. For each row record the following variables: CSS (on a scale 0 to 1), RaterID (say, 1 to 5), DocumentID (say, 1 to 100) and Relevant (binary, 1 if relevant, 0 if not relevant). Make sure the subjects do not see the CSS while they make the determination of whether the query is "relevant" to the reference document. STEP 4: Fit a statistical model that predicts a probability for relevant, P(Relevant). STEP 4A: One posibility is to go with a Logistic Regression model. Then you use "Relevant" as the response variable, and CSS as the predictor. (Ideally you may also want to first include RaterID, dummy-coded, using 4 dummy variables, as another predictor, and make sure it is insignificant; this will confirm that the rater has no effect, but you can skip this part if it is too much for your study). After you fit the Logistic Regression model, you will have an equation that produces predicted probability for "Relevant=1", given a CSS level. STEP 4B: A second possibility is to go with a decision tree model. This will create a set of if-then-else rules. All of them will have a predicted probability for "relevant" query on the right-hand side. STEP 5: Use your model to obtain predicted probabilities for all queries. Those cases where Pred.Probab. > 0.5 are the ones that you should classify as "relevant". Order your spreadsheet by increasing predicted probability and look up the point where the probability switvhes from less than 0.5 to greater than 0.5. Look up the corresponding CSS. This is your empirical CSS threshold, validated by human subjects, and estimated by a predictive model!

Good luck -- feel free to ask questions, such as what software to use, example code, etc.

Nick.

The main problem you have in this task is that a CSS threshold is a regularization parameter.

With it you control degree of generalization of the model.

If you make it too high, you will have high precision but low recall.

If too low, more entities will fall into "similar" set and there will be more mistakes.

The previous speakers told you the right way for choosing cut-off value.

Only your data has to determine the level of regularization if you don't have a priori knowledge about universe distribution.

Take a representative sample and tune CSS threshold by some optimization method for enough precision/recall ratio.

I want to turn attention on peculiar properties of CSS.

You get the same value for pairs of vectors that have equal angle between them irrespective of the vectors' norms.

So the CSS measure shows you high similarity for points that light years from each other.

Thanks guys (especially Nicholas) for all the answers and guidelines... All the best, Nikola

• but how to choose this alpha value based on vector space

Konstantin Nagaev , I'm sorry this is four year later but how do you calculate step 4 ( Fit a statistical model that predicts a probability for relevant, P(Relevant)? Thanks to anyone who can help!

Pascal Francq Do you have any reference for the approach you mentioned?

Not really. It seems a natural approach. I mentioned it in a scientific report for a research project a few years ago, but not in a reviewed article.

Hello Pascal Francq , thank you for your answers, what do you mean by alpha depends to the space vectors, can i choose it as a negative number ?

Thank you