Hello everyone,
I'm looking for a good algorithm to find similarity between documents.
I've cleaned up the text, performed summarization and extracted the most important words through tf-idf, entity extraction, and took those that are related to the previous one (NER) as well. The next step was stemming to get the final form. Which algorithm would you suggest to be the best for this? I've investigated a little bit and I found these:
1. Classic cosine similarity
2. Word Mover’s Distance
3. GloVe: Global Vectors for Word Representation
4. Siamese Manhattan LSTM (MaLSTM)
Thank you in advance :)
Thank you for the answer. I've been reading the whole day yesterday and I've decided to use word2vec because it allows me to find words that are semantically similar. What do you think about this? :)
What method of summarization are you using? It might already use some of your other techniques, and consequently bias your results in unexpected ways.
Also, I would do stemming first, not last.
When you will use Word2vec.
Suppose you have word X and words A,B,C. If you have word2vec model which contains all this words you can calculate distance from X to every of this words. Then you become to know that B is most similar to X, next is A, and after goes C. Are there any approaches to finding out that X is really close in meaning to B, or all A, B, C are very far semantically from X, although they are far away, they can always be ordered?
Thank you for your response Peter Ludemann.
I'm using the Centroid-based Text Summarization through Compositionality of Word Embeddings approach (the dilemma was whether to use TextRank or this one and I chose this, since it looks after diversity and coverage).
When it comes to the second part, I'm aware that some operations can be overlapped - I've cleaned the text and extracted all the data that I consider to be relevant. In practice, word2vec performs these procedures too. Having this in mind, I was looking more into using only specific functions (finding similarity) and NOT repeating what I've done before. I haven't tested it yet, so I don't know whether this is possible :)
Lastly, I planned to use Porter stemmer and in that case doing stemming first can be a real problem. Here's the scenario:
" For example, 'was' turns into 'wa' by porter stemmer and when you stemmed first before removing stopwords 'wa' remains in your vector after filtering stopwords which has 'was' as a stopword."
I'm still in a research process to find out how to do this on the best way I can.
NOTE: I'm pretty new to NLP, so my approaches might not be #1 thing to do but I read and consult with people doing NLP every day to improve myself.
Since I guess that this thread will be used by future newbies in NLP, I highly recommend starting with a good literature (not blogs, not courses). Why? Because they miss to point out that some cleaning processes (like stemming and lemmatization, etc.) are not default rules: each problem is specific and should be approached differently. Talk with experienced peers who can balance between advanced strategies and your level of knowledge - both of these approaches are the best starting point for me.
I switched to a different strategy than a this one. Thanks for everyone who contributed to this discussion :)
I recommend Schütze & Manning's books.
Porter stemmer isn't very good; you should be able to find something better.
Stop words are an optimization. TF/IDF will give low values to stop words anyway. (And stop words can create some corner cases, such as throwing away phrases like "to be or not be".)
Peter Ludemann I will check them and give a feedback after some time to share what kind of experience I had. Thank you for everything.
- Badges
- Science topic
- Similar topics
- Vectorization