I have a dataframe with 4 columns. Two columns are numerical, one column is text (tweets) and last column is label (Y/N). I want to convert text column into TF-IDF vector. Code for converting text into TF-TDF vector
from sklearn.feature_extraction.text import TfidfVectorizer
v = TfidfVectorizer()
x = v.fit_transform(df['tweets'])
Now i want to append the return document-term matrix into one of the new column of pandas dataframe, something like this
df['tweetsVect']=x
But this is not working..How to append this to new column of dataframe ?
There is a simple answer to this.
First. The reason for the error. The TF-IDF vectoriser produces sparse outputs as a scipy CSR matrix, the dataframe is having difficulty transforming this.
The solution is simple. Simply cast the output of the transformation to a list as follows:
df['tweetsVect']=list(x)
and this will store the data in a new column, but in a sparse format.
However, if you don't want the TF-IDF data stored in a sparse format, you can do something like:
df['tweetsVect'] = list(tweets_transformed.toarray())
which will add a dense representation.
Niall
Thanks a lot Niall. It works..Can i pass this dataframe column (tweetsVect) to classify a label (Y/N) ?
When i use this dataframe column tweets for classification, I am getting error "setting an array element with a sequence"
featureSet=df['tweetsVect']
labels=df['Status']
clfTaskCompletion = tree.DecisionTreeClassifier()
clfTaskCompletion.fit(featureSet, labels)
Hi Kumar,
Again, it's a pretty simple fix. Just call .tolist() on the feature and label set.
Below is a full working example:
Niall
---
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.tree import DecisionTreeClassifier
import pandas as pd
import numpy as np
df = pd.DataFrame({'tweets': ['one', 'two', 'one two three', 'four'], 'labels': [1, 0, 1, 0]})
vectoriser = TfidfVectorizer()
df['tweetsVect'] = list(vectoriser.fit_transform(df['tweets']).toarray())
tree = DecisionTreeClassifier()
tree.fit(df['tweetsVect'].tolist(), df['labels'].tolist())
Thanks Niall ! One more query
What if I have multiple column as feature set in my dataframe. I have used the same code, but it's giving error. I have added one more feature to the dataset and want to use that too for building classifier model. Snippet below:
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.tree import DecisionTreeClassifier
import pandas as pd
import numpy as np
df = pd.DataFrame({'tweets': ['one', 'two', 'one two three', 'four'],'labels': [1, 0, 1, 0], 'x':[5,10,15,25]})
vectoriser = TfidfVectorizer()
df['tweetsVect'] = list(vectoriser.fit_transform(df['tweets']).toarray())
#considering both feature x and tweetsVect
featureSet=df.iloc[0:,2:]
tree = DecisionTreeClassifier()
tree.fit(featureSet, df['labels'].tolist())
--
Thanks and regards,
Kumar Abhinav
Hi again Kumar,
This can be done with a little more work. Basically, you will want to look at sklearn.pipeline.FeatureUnion and sklearn.pipeline.Pipeline
There's a good example of doing this kind of thing on sklearn's documentation.
http://scikit-learn.org/stable/auto_examples/hetero_feature_union.html
Niall
http://scikit-learn.org/stable/auto_examples/hetero_feature_union.html
Thanks a ton Niall for all the help. Really appreciate your help. I used DataFrameMapper for my work. Also tried Feature Union and Pipeline. DataFrameMapper works for my problem.
https://github.com/paulgb/sklearn-pandas
--Thanks and regards,
Kumar Abhinav
Hi Kumar,
Can you please provide your solution because I can't solve the same problem!
Thank you very much
Luca
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python