Observants 1 and 2 both entered a total of 477 observations, divided over 10 cases. So an average of 47.7 measurements per case, where within each measurement the answer options were different (ranging between 2 and 30 options). I have counted that in 440 cases, the observations between observer 1 and 2 are equal. In 37 cases, the observers measured something different. Can I measure the total inter-rater reliability based on this data? How? I believe this does not correspond to the standard Cohen's Kappa cross table (yes / no / yes / no).
A percentage agreement between raters is the fundamental indicator for inter-rater reliability. Judges decided on three out of five ratings in this competition. The agreement ratio is 3/5 = 60%. A table (like the one above) is helpful in seeking a percentage agreement between two raters.
Hello Rob,
Your summary of absolute agreement between judges would suggest that the rate is 440/477 = 92%, which, in many scenarios would be considered good to excellent. Your threshold for characterizing the result may differ, of course.
There are other methods, but this is simple, straightforward, and unlikely to be misinterpreted.
Good luck with your work.
Mahadi Hasan Miraz Thanks. But nog not sure how I have to apply your explanation to my example. I think this is more or less the same as David Morse 's answer, right? Concerning your reply David Morse this is something i can relate to. I then assume this percentage is a measure with enough scientific value. (because with your adviced method I'm not accounting for for coincidental matching observations, which Kappa does)
Rob Schip I think you are smart enough. I believe youtube is enough for you. You are smarter than me. Please go ahead.
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology
- Research Papers