After extracting news from different news websites, I need to find the similarities between them. Which are the best techniques I can use for finding document similarity ? I'm planning to use Word2vec + cosine similarity measure so far. Word2vec for converting words into vector space and then apply cosine similarity matrix on that vector space. But I'm not sure I can use both of them together.
Firstly, Word2Vec is used to represent words.
To find the document similarity you have to use same concept for documents (Doc2Vec model ), then you can find the document similarity by calculating cosine similarity for docs vectors.
The following implementations and examples of using doc2vec model for document similarity:
https://praveenbezawada.com/2018/01/25/document-similarity-using-gensim-dec2vec/
http://ai.intelligentonlinetools.com/ml/text-clustering-doc2vec-word-embedding-machine-learning/
https://kanoki.org/2019/03/07/sentence-similarity-in-python-using-doc2vec/
https://www.programcreek.com/python/example/96043/gensim.models.Doc2Vec
Good day!
It is not necessary to use both techniques, they are rather different.
When you use word2vec you map text to the vector:
map:
I robot not
1 1 1
So, with this map:
I robot = 1 1 0
I not robot = 1 1 1
robot not = 0 1 1
I robot robot robot not = 1 3 1
The main idea of word2vec usage to find similarity between vectors, for example, by the sum of product:
EXAMPLE 1
I robot robot not = 1 2 1
I not robot = 1 1 1
Similarity = 1*1 + 2*1 + 1*1 = 4 (quite similar)
EXAMPLE 2
I robot = 1 1 0
not robot = 0 1 1
Similarity = 1*0 + 1*1 + 0*1 = 1 (quite not similar)
EXAMPLE 3
I = 1 0 0
not robot = 0 1 1
Similarity = 1*0 + 0*1 + 0*1 = 0 (not similar at all)
If you use cosine similarity, it will be rather different:
EXAMPLE 3: cosine = 0 (the sets are not intersect)
EXAMPLE 4: "I robot" and "I robot" cosine = 1 (they are the same set)
BUT! You can use word2vec and cosine together as weighted sum or maximum of their normalized values.
For example:
cosine in [0,1]
word2vec products in [0, 10]
Mixed similarity = cosine + word2vec / 10 (will be in [0,2])
Mixed similarity = ( cosine + word2vec / 10 ) / 2 (will be in [0,1])
Mixed similarity = max ( cosine, word2vec / 10 ) (will be in [0,1])
- Badges
- Science topic