I have considered 3 datasets and 4 classifiers & used the Weka Experimenter for running all the classifiers on the 3 datasets in one go.
When I Analyze the results, considering say classifier (1) as the base classifier, the results that I see are :
Dataset (1) functions.Linea | (2)functions.SM (3) meta.Additiv (4) meta.Additiv
--------------------------------------------------------------------------------------------------
'err_all' (100) 65.53(9.84) | 66.14(9.63) 65.53(9.84) * 66.14(9.63)
'err_less' (100) 55.24(12.54) | 62.07(18.12) v 55.24(12.54) v 62.08(18.11) v
'err_more' (100) 73.17(20.13) | 76.47(16.01) 73.17(20.13) * 76.47(16.02)
--------------------------------------------------------------------------------------------------
(v/ /*) | (1/2/0) (1/0/2) (1/2/0)
As far as I know:
v - indicates that the result is significantly more/better than base classifier
* - indicates that the result is significantly less/worse than base classifier
Running multiple classifiers on single database is easy to interpret, but now for multiple datasets, I am not able to interpret which is better or worse as the values indicated do not seem to match the interpretation.
Can someone pls. help interpret the above result as I wish to find which classifier performs the best & for which dataset.
Also what does (100) next to each dataset indicate?
'err_all' (100), 'err_less' (100), 'err_more' (100)
The methodology for comparing n machine learning (ML) methods over m data sets is described in (Demšar, 2006). Please see the link.
I show you an example in the attached document.
Article Statistical Comparisons of Classifiers over Multiple Data Sets
Hi Renu,
Generally, over 3 datasets, second and fourth classifiers have similar performance. And those classifiers have similar behavior to the first classifier when working on two datasets (1st and 3rd) only. On the other hand, second and fourth classifiers are statistically better than first one only on 2nd dataset. However, first classifier is statistically better than third classifier on two datasets (1st and 3rd). In contrast, third classifier is statistically better than first one when working on second dataset. In sum, since second and fourth algorithms almost have same performance, you many need to replace one of them with another different algorithm to avoid any sort of replication in the performance, or you could simply remove one of them. Moreover, third classifier is considered to have less performance than first one.
Nevertheless, for your second question, this is because you’re performing stratified 10-fold cross-validation 10 times (10*10=100).
HTH.
Samer
- Badges
- Science topic
- Similar topics
- Organizational Psychology
- Training