I trained a set of machine learning classifiers on a dataset, and I select the value of Kappa and ROC area from the classifer results,
Example:
Classifier ROC area Kappa Value
IBK 0.879 0.3452
SMO 0.962 0.7321
J48 0.985 0.4325
Random Forest 0.654 0.4952
Naive Bayes 0.875 0.8421
How Can I compute the P-value From ROc area and Kappa value tests?
Hi Khalid
Statistical significance for kappa is rarely reported, probably because even relatively low values of kappa can nonetheless be significantly different from zero but not of sufficient magnitude to satisfy investigators. So you can't, in general,reliably report the p-value for a kappa value.
Ror ROC area, it's easy to report the confidence intervals for sensitivity and specificity using bootstrapping.
My advice - which you should ignore if your specific investigations are non-standard - is to report kappa statistics with no p-values, and to report AUC with 95% CI for ROC curves.
- Badges
- Science topic