I am working on classification based problems using various supervised machine learning algorithms. I have found out simple accuracy based on correct classification. Is it necessary to go for statistical analysis like Wilcoxon sum rank test and t-test ? Is it of any real use?
Hi Manas,
If you use supervised models and you want to evaluate its performance you should calculate true positives rate (TPR) and false positive rate (FPR): sensitivity , specificity. For evaluation of your model you can than use Receiver operating characteristic (ROC).
Your question is really two questions which could be answered as follows:
1) One of the main reasons you should use them is that you used a data sample for your tests. Are they enough to prove your algorithm is good on the whole range of the dataset? could your test set be biased towards good performance? This is a difficult question to address with just an accuracy based measure.
2) Non parametric testing such as Spearman, Whitney, and Friedmann does not assume a particular distribution. This is especially important for complex datasets (almost all real life datasets).
For a good discussion on which method to use you can look at:Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7, 1-30.http://jmlr.org/papers/volume7/demsar06a/demsar06a.pdf
The use of statistical methods has been discussed for a very long time and has been largely ignored by the community, even though it is essential (regardless of the method you use). Some of the first references that address this problem are:
Prechelt, L. (1996). A quantitative study of experimental evaluations of neural network learning algorithms: Current research practice. Neural Networks, 9(3), 457-462.Flexer, A. (1996). Statistical evaluation of neural network experiments: Minimum requirements and current practice. Cybernetics and Systems Research, 1005-1008.Hope this helps
I believe it is in general good practice to test if the difference between the accuracy of two classifiers is statistically significant. To do so, however, you need to estimate their accuracy on a number of different 'problems', which you can obtain from different datasets or even from the same dataset by different subdivisions into train and test set.
Theoretically the t-test is not a good choice in this case: being bounded by 0 and 1 accuracy does does distribute normally (though it would be acceptable if the accuracy is far from the limits).
In any case I think it's better to stick to non parametric, rank-based tests (see Arturo's answer.)
Best,
f.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis