I'm new to python and NLP. I need to compute the similarity score of piratical text with many other tests and return the 5 top texts with a highest similarity.
The method that I need to use is "Jaccard Similarity ". the library is "sklearn", python. I have the data in pandas data frame. I want to write a program that will take one text from let say row 1 of column 3, and compared with all other text from other rows in column 3 and return similarity score. please help me in how to get started, I search a lot on google but very confusing.
If you just have a small number of texts, search with aunt google for ‘minHashing’. If you have millions of texts you should additionally search for ‘local-sensitivity hashing’. Reading the corresponding chapter in ‘Mining Massive Databases’ might be helpful.
Here is your answer.
- Jaccard similarity
import pandas as pd
all_documents = pd.read_csv(path/to/csv)
#assume all_documents contains string in all columns
tokenized_row_columns3 = [d.split() for d in all_documents['column3'] #list
tokenized_row_columns2 = [d.split() for d in all_documents['column2'] #list
tokenized_row_columns1 = [d.split() for d in all_documents['column1'] #list
def jaccard_similarity(query, document):
intersection = set(query).intersection(set(document))
union = set(query).union(set(document))
return len(intersection)/len(union)
call the jaccard_similarity as:
#for similarity of 1 and 2 of column1
jaccard_similarity(tokenized_row_columns1[1],tokenized_row_columns1[2])
#for similarity of row1 of column1 and row1 of column3
jaccard_similarity(tokenized_row_columns1[1],tokenized_row_columns3[1])
ALTERNATIVE
- Cosine similarity in Scikit-Learn
Assume you are focusing on column 3.
Let assume the rows of column 3 are documents.
Use column 3 to create tfidf.
You can see tfidf of row/document 1 as tfidf[1].
Then proceed to similarity score as:
similarity(tdidf[0],tfidf[1]) # similarity of row/document 0 and row/document 1
Yes you can create separate tfidf for each column and continue with similarity.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import pandas as pd
all_documents = pd.read_csv(path/to/csv)
sklearn_tfidf = TfidfVectorizer(max_features=20000)#choose your own max
#tfidf of column3
column3_tfidf = sklearn_tfidf.fit_transform(all_documents["column3"])
#cosine similarity of row/doc 1 and row/doc 2 of column3
cosine_similarity(column3_tfidf[1], column3_tfidf[2])
Now create tfidf for other columns as:
column2_tfidf = sklearn_tfidf.fit_transform(all_documents["column2"])
column1_tfidf = sklearn_tfidf.fit_transform(all_documents["column1"])
#cosine similarity of row1 of column1 and row 2 of column2
cosine_similarity(column1_tfidf[1], column2_tfidf[2])
Further info here
http://billchambers.me/tutorials/2014/12/21/tf-idf-explained-in-python.html
Thank you for your help. In order to compute the Jaccard similarity we need to tokenize the texts. In the example that you mention all the documents are putted in list. Which it dose not work in my case as I have all the texts in the rows of column 3. I'm new to python , I have pandas data frame, table of columns and rows. Each row in column 3 contain a long text. I need to compute the Jaccard similarity of each row with all other rows, and return the top 5 most similar ones. I'm stick in how to tokenize the texts in these rows of pandas data frame. Is there are any library to do so!. please help. Thanks in advance
Look at my answer, for example
tokenized_row_columns2 = [d.split() for d in all_documents['column2']
In simple form it means.
tokenized_row_columns2=[] # an empty list that will contain list of list
for d in all_documents['column2']: #a for loop to loop on all rows in column2
#d is a row (of type string).
#Then split() will tokenize the row.
#e.g "I am wise".split() will be a list of tokens["I","am", "wise"]
token = d.split() #token is a list of tokens
tokenized_row_columns2.append(token) #add tokens
#end for loop
AT the end you will have list of list. e.g.
tokenized_row_columns2 = [[ 'row1', 'of', 'column2' ] ,[ 'row2', 'of' 'column2' ],...]
tokenized_row_columns2[0] is [ 'row1', 'of', 'column2' ] #list of tokens in row1
tokenized_row_columns2[1] is [ 'row2', 'of' 'column2' ] #list of tokens in row2
#for similarity of row 0 and 1 of column2
jaccard_similarity( tokenized_row_columns2[0], tokenized_row_columns2[1] )
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python