Hello,
I have built a naive bayes classifier in java in order to help sort through publications for a meta analysis project. When developing the classifier I used an available training set that consisted of positive and negative reviews and it was able to predict with about 90% accuracy which ones were which. However now that I transition data sets to a data set more representative of what my "real" data set will be (mostly useless articles with a couple relevant articles scattered throughout the set) I am having some trouble. I am still predicting with high accuracy (~96%) but it never classifies any of the articles as being relevant. I am guessing because statistically it will always make sense to guess negative because that is by far what is most abundant. Is there any way to deal with this problem? Any suggestions would be helpful. Thanks!
Found a paper that discuses possible remedies to this problem -> https://www.cs.waikato.ac.nz/~eibe/pubs/FrankAndBouckaertPKDD06new.pdf
kindly use R, nowhere will be easier than this
This is merely a side note because it does not solve the actual problem: In binary classification problems with unbalanced data sets such as yours accuracy is not a good measure since, just as you mentioned, accuracy can be very high without any skill of the model. The Receiver Operating Characteristic (ROC) or the Matthews Correlation Coefficient are less vulnerable and allow for better interpretation of the model skill.
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training