I am working on increasing inter-rater reliability for a video coding project, and my advisor and I came to the conclusion that a weighted kappa would be the appropriate measurement to use (raters are watching video clips and rating levels of different attitudes/behaviors on a 1-5 scale, so an ordinal measure). I installed the spss extension to calculate weighted kappa through point-and-click.
But, when checking through an example, I got Kappa scores that were inconsistent with what I thought would happen, and I am wondering if the program is wrong, or if my understanding is off (and this measure may be unfit). I attached a picture to show the rating scores from 3 raters (A, M, & E) for a variable, and the kappa output I received for each pairwise comparison. When comparing the other raters to A, both M and E only disagree with A once, and each only a 1-point difference away. But, the weighed kappas paired with A are different, it appears because the 1-point difference is with a higher or lower number (4/3 versus 2/3). Is this correct? Shouldn't weighted kappa consider all 1-point differences equally and just considering whether it's 1/2/3 numbers away for the reliability?
A weighted version would precisely take into account the baseline occurrence of ratings when appraising relative-differences.
Hello Emily,
The computations of the weighted kappas are correct as given in the SPSS add-in. The reason that they don't agree for your two cases (A, E; A, M) is that the two sets have different marginal distributions. Specifically, Friend E and Friend M have different marginal distributions:
Friend E: 2) 0 3) 2 4) 2
Friend M: 2) 1 3) 2 4) 1
[FYI, Friend A: 2) 1 3) 1 4) 2 ]
Therefore, when paired with Friend A, different expected cell frequencies are generated for the two pairings (under the hypothesis of no relationship).
See this site for a more detailed explanation: http://www.real-statistics.com/reliability/weighted-cohens-kappa/
Good luck with your work!
Dear Emily. Interesting question. I wonder why did you decide weighted Kappa, rather than Cohen kappa. Why did you choose to use a nominal 1-5 scale? If you are using criteria to assess the accuracy of video, say if the item was fulfilled, you give 1 or if not you will give zero. In this situation, the inter-rater agreement between evaluators should be about scores, rather than ordinal measures.
David Morse, Thank you, that definitely helps clear it up, and the link helps as well. Though it also highlights that reaching a great reliability will be more difficult than I'd hoped, but that's research for ya.
Samy Azer, to clarify my reasoning on a weighted kappa: we are using ordinal Likert scales, where a 1 rating means very low or none and 2-5 indicate increasing degrees of the variables (but they are not necessarily evenly spaced). Heyman and colleagues' chapter on behavioral observation and coding recommended checking coder reliability for these types of measures using Finn's r or weighted kappa, and my advisor has some experience with weighted kappa in the past. (http://psych.nyu.edu/westlab/documents/West_Chapter_2014_Behavioral%20observation%20and%20coding.pdf)
- Badges
- Science topic
- Similar topics
- Cognitive Science
- Thinking