Hi. Are there any merits of one over the other for evaluating classifiers (e.g. SVM for image processing problem domain)?
Neither! And both! :-)
Confusion matrices by themselves do not do anything. By taking combinations and ratios of confusion matrices rows/columns and their marginals, you extract lots of useful performance metrics, including accuracy, precision, recall, f-measure, etc. Each of these has their own merits, and there are occasions where one should be preferred over the other. For example, if there is a significant class imbalance, you should prefer to assess performance with f-measure over accuracy, as accuracy can give a misleading belief about performance. (Read the first link for more "The case against accuracy estimation for comparing induction algorithms".) So, confusion matrices are not very useful, but they can certainly help you understand the nature of performance metrics.
But, to get the confusion matrix in the first place you will have needed to select a threshold somewhere along the line. There are a number of approaches for selecting thresholds, and ROC analysis is quite a powerful option. ROC analysis is particularly useful for threshold selection if your classes each have different misclassification costs, e.g. in medical fields false positive classifications are more tolerable than false negatives. ROC analysis can easily let you do cost-sensitive threshold selection that will let you select your threshold optimally for accuracy. (In a related note, there has been some recent work that allows similar threshold selection for f-measure, see "Precision-Recall-Gain Curves: PR Analysis Done Right", link #3 below.)
The area under ROC curves (AUROC) can also tell you something about the classifier. You often read that the AUROC is the probability that any randomly selected positive example will have a higher score than a randomly selected negative sample. So with this interpretation, AUROC is related to ranking, and this doesn't have an obvious link to accuracy yet. It can be shown, though, that the AUROC is related to accuracy in the following manner:
[math]\mathbb{E}(acc) = \frac{AUROC}{2} + \frac{1}{4}[/math]
for even misclassification costs/class distributions, and in this manner we can see the relationship between the two. For more information about ROC/AUROC analyses, I recommend reading "An introduction to ROC analysis" by Tom Fawcett, see the second link below.
So, what I'm trying to say is that performance evaluation is complicated. Depending on the problem you are working on you should prefer one evaluation method over others.
(Note, the comments below are not limited to just image processing.)
http://eecs.wsu.edu/~holder/courses/cse6363/spr04/pubs/Provost98.pdf
http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf
http://papers.nips.cc/paper/5867-precision-recall-gain-curves-pr-analysis-done-right
Neither! And both! :-)
Confusion matrices by themselves do not do anything. By taking combinations and ratios of confusion matrices rows/columns and their marginals, you extract lots of useful performance metrics, including accuracy, precision, recall, f-measure, etc. Each of these has their own merits, and there are occasions where one should be preferred over the other. For example, if there is a significant class imbalance, you should prefer to assess performance with f-measure over accuracy, as accuracy can give a misleading belief about performance. (Read the first link for more "The case against accuracy estimation for comparing induction algorithms".) So, confusion matrices are not very useful, but they can certainly help you understand the nature of performance metrics.
But, to get the confusion matrix in the first place you will have needed to select a threshold somewhere along the line. There are a number of approaches for selecting thresholds, and ROC analysis is quite a powerful option. ROC analysis is particularly useful for threshold selection if your classes each have different misclassification costs, e.g. in medical fields false positive classifications are more tolerable than false negatives. ROC analysis can easily let you do cost-sensitive threshold selection that will let you select your threshold optimally for accuracy. (In a related note, there has been some recent work that allows similar threshold selection for f-measure, see "Precision-Recall-Gain Curves: PR Analysis Done Right", link #3 below.)
The area under ROC curves (AUROC) can also tell you something about the classifier. You often read that the AUROC is the probability that any randomly selected positive example will have a higher score than a randomly selected negative sample. So with this interpretation, AUROC is related to ranking, and this doesn't have an obvious link to accuracy yet. It can be shown, though, that the AUROC is related to accuracy in the following manner:
[math]\mathbb{E}(acc) = \frac{AUROC}{2} + \frac{1}{4}[/math]
for even misclassification costs/class distributions, and in this manner we can see the relationship between the two. For more information about ROC/AUROC analyses, I recommend reading "An introduction to ROC analysis" by Tom Fawcett, see the second link below.
So, what I'm trying to say is that performance evaluation is complicated. Depending on the problem you are working on you should prefer one evaluation method over others.
(Note, the comments below are not limited to just image processing.)
http://eecs.wsu.edu/~holder/courses/cse6363/spr04/pubs/Provost98.pdf
http://people.inf.elte.hu/kiss/13dwhdm/roc.pdf
http://papers.nips.cc/paper/5867-precision-recall-gain-curves-pr-analysis-done-right
Hello
I like Niall's answer and just gave it a thumbs up. Traditionally, I mean for detection theory, the ROC gives the full locus of PD and PFA for all possible thresholds in the hypothesis test. A confusion matrix, as previously mentioned, is for a given PD and PFA. There is some evidence that the area under the ROC curve is a more useful measure than accuracy than the usual ratio measures.
Thank you Niall for your detailed answer. And thanks to Graham and Waldemar for your responses too.
I will go through the publications that you have linked.
I guess as Niall has mentioned, it is neither and both. And very much dependant on the context and problem domain.
Thanks
Ahmed
- Badges
- Science topic
- Similar topics
- Engineering
- Electrical Engineering
- Signal Processing
- Image Processing