If we have for two experiments (EXP1 and EXP2), the same accuracy rate and the following results:
EXP1:
a precision of 0.652
F-measure = 0.551
ROC = 0.74
Recall = 0.567
vs.
EXP2:
a precision of 0.521
F-measure = 0.529
ROC = 0.854
Recall = 0.567
As you can see we have different precision rates and ROC areas!
Which of these results is better than the other and why? (The best experience =?)
Can the ROC area be more relevant than precision as a performance measure in a given classification system? Thanks in advance.
I think the following article would be helpful to you in this regard:
Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation
DM Powers - 2011 - dspace2.flinders.edu.au
ROC curve is more reliable, specially when you are going to compare your method with the other available methods. The key difference is that ROC curves will be the same no matter what the baseline probability is. Precision is a probability conditioned on your estimate of the class label and will thus vary if you try your classifier in different populations with different baseline.
Moreover, when you are testing a machine learning system, and it is score based, ROC all the times is the same, but by selecting different threshold level you can achieve different precision values.
Classification Assessment Methods
This paper introduces a detailed explanation with numerical examples many classification assessment methods or classification measures such as:
Accuracy, sensitivity, specificity, ROC curve, Precision-Recall curve, AUC score and many other metrics. In this paper, many details about the ROC curve, PR curve, and Detection Error Trade-off (DET) curve. Moreover, many details about some measures which are suitable for imbalanced data are explained.
Your comments are highly appreciated.
https://ac.els-cdn.com/S2210832718301546/1-s2.0-S2210832718301546-main.pdf?_tid=20353999-8c84-422a-8191-fd1059469272&acdnat=1536185805_03b19e159f0675a01f35512961973eb5
Presentation of the paper is here
https://www.researchgate.net/publication/327403649_Classification_assessment_methods_a_detailed_tutorial
- Badges
- Science topic