I am working on Physical Activity Recognition using data acquired from smartphone sensors (gyroscope, magnetometer and accelerometer) and would like to compare different classifier performance on the dataset, but wondering which evaluation matrix would be best to use: True Positive Rate (TPR), False Positive Rate (FPR), Precision, Recall, F-score or overall classification accuracy? This is a six class problem (6 different activities).
Rajeev.
TPR and FPR separately do not tell you anything. The same goes for Precision and Recall. F-score, Equal Error Rate and similar can give you some information, but it is limited and you have to be careful how you use those values.
Look at this previous question:
https://www.researchgate.net/post/How_do_we_tell_which_classifier_does_the_best_detection_job
I'll copy part of my answer for that question here:
You have to first answer a simple question: Do You know what are the costs of the decisions?
If the costs are known, you can calibrate your classifiers (set the operating point), and compare costs of the classifiers.
If you do not know the costs, you may have some insight about a reasonable range of operating points of the target application. Lets say the target application is a surveillance system, and all positive responses will have to be shown to a human operator which is able to process one event per minute. In such case, it makes sense to compare detection rates at one false positive per minute. You can use other measures instead, but it makes sense to keep one type of error fixed and compare the other one.
If you have no knowledge about the possible operating point in the target application, you should show results in full range of possible operating points. Plot ROC, precision-recall, DET or something similar. You can also compute area under ROC or Precision-recall and get a numeric value which reflects performance over the whole range of regimes; however, such aggregation has its own problems.
Rajeev.
TPR and FPR separately do not tell you anything. The same goes for Precision and Recall. F-score, Equal Error Rate and similar can give you some information, but it is limited and you have to be careful how you use those values.
Look at this previous question:
https://www.researchgate.net/post/How_do_we_tell_which_classifier_does_the_best_detection_job
I'll copy part of my answer for that question here:
You have to first answer a simple question: Do You know what are the costs of the decisions?
If the costs are known, you can calibrate your classifiers (set the operating point), and compare costs of the classifiers.
If you do not know the costs, you may have some insight about a reasonable range of operating points of the target application. Lets say the target application is a surveillance system, and all positive responses will have to be shown to a human operator which is able to process one event per minute. In such case, it makes sense to compare detection rates at one false positive per minute. You can use other measures instead, but it makes sense to keep one type of error fixed and compare the other one.
If you have no knowledge about the possible operating point in the target application, you should show results in full range of possible operating points. Plot ROC, precision-recall, DET or something similar. You can also compute area under ROC or Precision-recall and get a numeric value which reflects performance over the whole range of regimes; however, such aggregation has its own problems.
Hi,
I would just like to point out some more (exotic?) measures which might be used. The MCC (see link) is probably the most suitable if you don't have any costs as Michal explained. It is also very easy to compute and it considers the whole confusion matrix.
As the question is very general, I'd also mention that In some cases you might want to use Fischer's exact test or Barnard's exact test to get the significance of the results (i.e. how likely it is that the classifier gave that result by chance).
Cheers,
Zoltan
http://en.wikipedia.org/wiki/Matthews_correlation_coefficient
Thanks Zoltan Marton...This is a good method to compare the performance of two classifiers with statistical tests. For some time i have been using 5x2 cross validation paired t-test as outline in the following journal:
http://www.mitpressjournals.org/doi/abs/10.1162/089976698300017197?journalCode=neco#.U3GdQnaleJw
Will also try out the ones you have outline above. Thanks once again
Other alternative is the Kappa index which would be a good complement with ROC curve, TPR, and FPR. For definition, the Kappa index is a statistical measure that consider inter-rater agreement and inter-annotator agreement based on categorical items. You can associate the categorical items with the 6 physical activities (given by the classifier) and to determinate the quality of the classification. In some applications ROC curve, TPR, FPR, and the Kappa index are enough to find the best classification. However, I suggest that you seek the validation index for tasks of classification but they are useful in unsupervised classification although it is other strategy to focus on supervised classification, if it is possible.
- Badges
- Science topic