The Matthew Correlation Coefficient (MCC) is often used to measure the performance of a binary classifier. If you have two such values - one trying to classify a class that occurs 10% of the time, and another trying to classify a class that occurs 20% of the time - can you directly compare those values? Or do you need to do some sort of skew-normalization?
http://en.wikipedia.org/wiki/Matthews_correlation_coefficient
I am a little lost in your question. As I see it, accuracy (however measured) is a highly domain-dependent characteristic, so it makes sense as long as you compare it to some base rate (e.g. an algorithm that guesses at random). MCC is probably one of the stabler measures with respect to class imbalance, but my doubt remains: why would you compare accuracies between different datasets?
If you have a multi-class prediction problem, then you should switch to some multi-class measure of accuracy. Instead, if you are comparing several algorithms on different datasets, you should probably perform a statistical testing (e.g. the Friedman test).
Thanks for the response Simone. My question was based on the conclusions of the paper linked below. Based on this supplemental figure from the same authors, it looks like MCC is also sensitive to skew, albeit less so than some others (at least when the ratio of false positives to false negatives is balanced).
http://www.pitt.edu/~jeffcohn/skew/PID2829477.pdf
Hi Jeffrey,
You are right MCC is sensitive to unbalanced data to a lesser extent than for example accuracy (num of correct predictions/total num of predictions). There is also the kappa index, in addition have you considered using specificity X sensitivity as an accuracy measure? By multiplying these two measures it will give you an overall accuracy and prediction ability of two classes (see my attachment - it also includes some references that might help!).
However the point you are trying to make is comparing two data sets (even if the compounds/predictor are the same) as by having different distributions of classes (you said 10% and 20%) will result in different classification problems and potentially different models therefore are not really comparable. However if you had to make a comparison you would have to make clear that these are not really comparable as they are separate classification problems in their own right.
Hope that helps a little!
Article Coping with Unbalanced Class Data Sets in Oral Absorption Models
- Badges
- Science topic