Instead of using accuracy, I'm trying to find which is the best classifier based on precision, recall and f-measure. I have the idea what they are individually, but can't really explain the results when they come together.
In the table for example, do we say BN is the best classifier because of its highest precision? But it has lowest recall and F-measure. Or do we simply say DT, IBL and MLP performed better because they have consistent results etc etc?
Helo Aida,
Correlation Coefficient in statistic is a measure of how well the relationship between two variables can be described (see http://en.wikipedia.org/wiki/Correlation_coefficient).
In machine learning if you have a supervised system the two variables would be:
1- expected values
2- values predicted by the classifier
and the Correlation Coefficient (Sperman, Pearson, etc) is a measure that increases as the classifier was able to better predict the expected results.
Analysis of classification result can be done in more than one way, and it depends on your problem. In some cases, you can live with lot of false-positives, but false-negatives is a sure no-no (like medical problems). In some cases both false-positive and false-negatives are treated equally bad. Choose sensitivity, specificity limits, accordingly.
In certain multi-class problems, accuracy can be a good global measure. However, accuracy chosen should take into consideration class imbalance, class-wise sample distribution (combination of local to get global measures). One can look at average accuracy, geometric mean accuracy (for ex. as defined in dx.doi.org/10.1109/TFUZZ.2013.2242894), which considers class-wise classification accuracy in deciding total score.
Hi. I agree with Kartick Subramanian, he is spot on. I'll just elaborate bit more on binary classification which seems to be your topic.
You have to first answer a simple question: Do You know what are the costs of the decisions?
If the costs are known, you can calibrate your classifiers (set the operating point), and compare costs of the classifiers.
If you do not know the costs, you may have some insight about a reasonable range of operating points of the target application. Lets say the target application is a surveillance system, and all positive responses will have to be shown to a human operator which is able to process one event per minute. In such case, it makes sense to compare detection rates at one false positive per minute. You can use other measures instead, but it makes sense to keep one type of error fixed and compare the other one.
If you have no knowledge about the possible operating point in the target application, you should show results in full range of possible operating points. Plot ROC, precision-recall, DET or something similar. You can also compute area under ROC or Precision-recall and get a numeric value which reflects performance over the whole range of regimes; however, such aggregation has its own problems.
I hope this helps. Regards.
Here are some papers you might find informative. The first discusses how to compare classifiers. The second discusses problems with doing so.
Assessing the accuracy of prediction algorithms for classification: an overview
http://bioinformatics.oxfordjournals.org/content/16/5/412.full.pdf
Classifier Technology and the Illusion of Progress
http://arxiv.org/pdf/math/0606441.pdf
Thanks all for the information! I learnt that accuracy is not the best measurement for imbalance dataset and that I should use precision and recall. In general, precision has inverse relation with recall and F-measure is the mean of both (I know this is oversimplified). So, when we present the results for precision/recall (I guess we choose which based on Jin's answer), do we need to explain the F-measure as well?
Aida, what you wrote would probably lead to completely wrong approach. You can't compare just recalls or just precisions. You can't even compare classifiers which have fundamentally different operating points. You'll get meaningless results in both cases.
I see, thanks Michal. I will go back reading papers suggested by Michael first.
- Badges
- Science topic