Hello,
I am developing 2 AI models and for both models I have compared the models predictions to that of 5 raters. I have calculated the cohen's kappa coefficient between each rater and model1 and then calculated a mean kappa (or median still unsure how to present this?) value for this model. I then repeated this process for model2.
So now I have 2 average cohen kappa coefficient values:
Average rater-model1 kappa values + SD
average rate-model2 kappa value + SD
How do I determine if these 2 kappa values are statistically significantly different?
Hello Oreoluwa,
If you're confident in considering the two values as independent terms, then a simple z-test statistic would be:
z = (mean1 - mean2) /SE
where SE = sqrt(1/n) * sqrt((sd1^2 + sd2^2)/2)
and n would be the number of values that were averaged to get the mean (= 5).
Good luck with your work.
Cohen’s kappa represents the proportion of agreement between raters/observers, accounting for chance agreement. The approximate sig value of the kappa coefficient does not suggest statistically significant agreement; rather, it indicates that the kappa coefficient is statistically significant from zero. Concerning the statistical significance of correlated agreement coefficients, you could check out the linearization method by Gwet (2016), fully cited below.
Gwet, K. L. (2016). Testing the difference of correlated agreement coefficients for statistical significance. Educational and Psychological Measurement, 76(4), 609–637. https://doi.org/10.1177/0013164415596420
Good luck,
I presume the kappa values are paired observations. That is, the kappa for Model-1for Rater A should be matched with the kappa for Model-2 for Rater A.
- Badges
- Science topic
- Similar topics
- Correction