In imbalanced class classification task, accuracy is not good enough as a main evaluation. There are several alternative evaluation, i.e.: AUC, kappa coefficient and Matthew's correlation coefficient (MCC). Amongs these three evaluations, which one is more effective and robust in imbalaced class classification task?
Well, I would remedy the problem by increasing the under-represented class instead of voting for the best evaluation metric. If, for some reason, you cannot do that, then the confusion matrix, AUC, and/or Cohen’s kappa are my way out.
I just read a note in the literature that confirms the opinion that AUC is a good metric to report for imbalanced data, see the book entitled: "Evaluating Learning Algorithm" by Nathalie and Mohak, 2011, p129.
Personally speaking, I would compute all the criteria and just list them in the table. For one specific task, choosing only one evaluation would result in bias any way...
Hellinger-distance decision trees are designed for imbalanced classification, and the empirical evidence is that they're state of the art.
My previous comment still holds but I thought of trying, herein, to enrich this discussion by adding the following remarks triggered by the above answers:
(1) I still recommend using the CM and AUC for they give a better statistical picture
(2) If you have the option to include various metrics, then that would be better as suggested above, else, and as per your question, (1) is my answer.
(3) If you have the option to choose which machine learning method to use, then decision tree learning methods (be it random forests, HDDT or alike) are less prone to imbalanced data sets.
(4) If you have the option to augment the data then that will eventually balance it.
(5) Depending on the context, maximizing/minimizing different metrics can be favored. For instance, in the area of credit card fraud detection (where dealing with imbalanced data is the norm), banks would be more interested in minimizing the FPR value.
I recommend to use the followings criteria (i.e, confusion matrix, AUC, F-measure) other criteria can be used such as
Kappa statistic
Mean absolute error
Precision
Recall
F-Measure
It isn't enough to use some single measure as sensitivity, specificity, precision, PPV, NPV, etc. To use complex criteria as AUC and F_measure also are not good. For details please read my note : Negative Results Is it Good to use F-measure on Classification Tasks?
- Badges
- Science topic
- Similar topics
- Data Mining
- Classification