Hi,
I am developing a research protocol to test the inter-rater reliability of a measurement tool for learning. I see that the minimum acceptable kappa value could differ from discipline to discipline depending on the degree of inter-rater agreement required? for instance, a higher degree might be needed in medicine/disease diagnosis than in education?
Hi Elias, I hope it serves you, this instrument is to evaluate how the student is in terms of skills before entering the contents of the study program, with this instrument I evaluate at the beginning of the school year
https://www.academia.edu/43917820/Evaluaci%C3%B3n_diagn%C3%B3stica_de_habilidades_de_pensamiento_e_informacionales_a_trav%C3%A9s_del_dise%C3%B1o_y_aplicaci%C3%B3n_de_tres_instrumentos_para_estudiantes_de_primer_grado_de_educaci%C3%B3n_primaria
Hello
The kappa coefficient describes the agreement between the two evaluators in the research. The amount of agreement should generally be expressed as a percentage and does not make a difference in the fields.
Thanks all for the insights. Ehsan Alinia Bengar but what amount/percentage of inter-rater agreement is enough to say a tool is reliable?
dear Mr Elias Damtew Assefa ,
Kappa values between (0.40) and (0.75) or 40% and 75% are acceptable.
Acceptable values of kappa depend on the number of categories (or codes, or ratings), and their probabilities. See:
Bakeman, R: (2022). KappaAcc: A program for assessing the adequacy of kappa. Behavior Research Methods. doi: 10.3758/s13428-022-01836-1
Bakeman, R., & Quera, V. (2011). Sequential analysis and observational methods for the behavioral sciences. Cambridge University Press.
Bakeman, R., Quera, V., McArthur, D., & Robinson, B. F. (1997). Detecting sequential patterns and determining their reliability with fallible observers. Psychological Methods, 2(4), 357–370. https://doi.org/10.1037/1082-989X.2.4.357
- Badges
- Science topic
- Similar topics
- Psychology
- Mental Processes
- Learning