I want to classify the news headline data. I am able to to make corpus , cleaning the data , train the data using SVM ( but only for small data set ) . I am not splitting the data into train and test data inspite i am using different set for test data ( but from headline data only).
I am able to train the model but while testing with test data .
Error: No. of Variables in both are different is coming.
Random forest ( Same Error)
I have tried Naive Bayes ( Accuracy is coming very less aprox 10%)
Dear Garg,
If new headlines are related to positive or negative sentiment of any stakeholders, you can do text classification using sentiment analysis. R studio is good in this task; but I suggest you to use Python better better handling the problem.
Its better to use large scale data set to learning process has been complete
in various scales....
Dear Garg,
If new headlines are related to positive or negative sentiment of any stakeholders, you can do text classification using sentiment analysis. R studio is good in this task; but I suggest you to use Python better better handling the problem.
Can you use glove vectors for classifying. I am not sure you will get them for R ..
In python we have gensim to convert Glove vectors to word2Vec
After you do that use xgboost .. this will be good.
One more question what is your corpora size ?
In line 55 of you script, you should pass the list of words which appears in the training set a dictionary parameter:
tdtm= DocumentTermMatrix(tcorpus,control=list(dictionary= dtm$dimnames$Terms))
and you should delete line 56 (tdtm=removeSparseTerms(tdtm,0.999))
then, you test set will have the same terms as in the training set.
Ronaldo Cristiano Prati
@Ronaldo Thank You So much...
Sandeep Kumar Mohapatra
@Sandeep I am trying to convert my data using word2vec in R
I am using Wiki Article dumps ...(.gz2 file) total of 14 GB (compressed)
i am falling short of Memory size(4GB)....can you suggest way for handling this or how to convert the wiki dumps to word2vec in parts and then later on combine the parts to get one single file for vectors
Please Help
- Badges
- Science topic