I have a training set of ca. 2500 manually categorized abstracts and want to automatically categorize 4M papers. What is the best tool to download? Till now, I have only 90% with Kappa at 0.8 in cross validation. I used NaiveBayesMultinomial on 2500 features -- statistics of most frequent words. As preprocessing, I removed stop words and stemmed the text.
There is even a theory that determines the quality of the SVM result: Thorsten Joachims' Text Categorization Model! His book on text classification is still the best regarding this topic. For practical reasons, you could use the RapidMiner text Mining extension which reads in texts and does all the transformations you want - just easy!
Support vector machines have shown to be quite good for that (although they are slow....). Do you apply any weighting scheme on the feature values (such as TF-IDF)?
Update: I can recommend to use Weka. It provides quite good support for text classification tasks and you can implement protoypical solutions quite fast. A simple example can be found at http://weka.wikispaces.com/Text+categorization+with+WEKA
Hi guy! Thank you a lot for the answers!
My email is [email protected], just contact me and I will send you the data directly over email.
My corpus is a bibtex-file with manually picked out "Behavioral Research" papers. Every bibtex-entry has an abstract in it. The rest of papers is from all other possible fields.
"Behavioral Research" for me is research made on behavioral data of humans and higher primates. This is behavioral economics, psychology, social science and even computer science dealing with social networks. I exclude papers dealing with cognitive abilities or papers based on surveys. Many papers investigate economic behavior of drugged subjects - such papers get often misclassified as medical ones. Then, there are papers, which are pure analytical game theoretical investigations without any data of human behavior - those are excluded. I exclude macroeconomic papers, which deal with inflation and stocks.
My best performance in cross validation is achieved by the code of NaiveBayesMultinomial included in Weka library. I did try out Support Vector Machines - the performance is lower. I count the frequencies of usage of ca. 2500 most used stems in every abstract. In this way, I get numeric input vectors with a boolean target value.
So, I can try weighing and TF-IDF as Sebastian suggested. And thanks to Ashish for the relevant literature.
Thank you Rustam Tagiew I recieve your bib. I will try to classify it automatically
As the others said, SVM works very well. I would reccomend LibSVM, too.
From my experience the Naive Bayes Multinomial outperforms the SVM in terms of accuracy and is more robust if you apply a feature reduction.
@kristijan, is there a specific feature reduction technique that you are refferring to?
I will also recommend LapSVM, for some datasets its performance is better than standard SVM.
@Samah: I did a benchmark with four classifiers(NBM, SVM, dec.-tree and k-NN) on text data, one year ago. I did feature reduction with Information-Gain-method and Bi-Normal-Seperation(by G.Foreman). With IG-method accuracy between NBM and SVM is similar, but slightly better for NBM, especially when using ntf- and tfidf-TDM. With BNS-method performance is same with binary TDM, but clearly better for NBM-classifier when using the other two TDMs . So NBM is more flexible in this case. As far as i remember quick tests with other reduction techniques had similar results.
Regarding SVMs: Search for "Transductive Inference for Text Classification
using Support Vector Machines." by T. Joachims (old but usefull).
LibLIN(afaik available in weka) is a library for performance optimized linear SVMs.
There is even a theory that determines the quality of the SVM result: Thorsten Joachims' Text Categorization Model! His book on text classification is still the best regarding this topic. For practical reasons, you could use the RapidMiner text Mining extension which reads in texts and does all the transformations you want - just easy!
RapidMiner (open source, download free) has a text processing extension with all the preprocessing (stemming, Bag Of Words with tiff or binary, etc.) and then you can run SVM and get the results. It is just drag and drop and the system takes care. Try it!
- Badges
- Science topic