I'm trying to calculate the kappa coefficient for inter-rater reliability analyses. I've found that if a column for one rater has all of the same answers (e.g., all zeros), then the kappa coefficient will not be calculated. The following error message appears: "No measures of association are computed for the crosstabulation of x* y.
At least one variable in each 2-way table upon which measures of association are computed is a constant."
Does anyone know how to get around this?
If there is no variability in the ratings there is no information from that rater so the only solution, as far as I'm aware, is to drop that rater.
Hello Jeremy,
Cohen's Kappa is a proportion agreement corrected for "chance level" agreement across two categorical variables. Kappa is defined as (Po - Pc) / (1 - Pc), where Po is the observed degree of agreement (as a proportion, from 0 to 1), and Pc is the chance-level agreement (derived from marginal frequencies and total N).
Here's an example, where rater 1 assigns all zeros and rater 2 assigns 60% zeros:
Rater 2: 0 1
Rater 1: 0 30 20
1 0 0
Po = 30 / 50 (the sum of 0-0 matches and 1-1 matches) = .60
Pc = 50*30/50^2 + 0*20/50^2 (row1 total*col1 total / N^2, + row2 total*col2 total/N^2) = .60 + .00 = .60
So, kappa will be: (.60 - .60) / (1 - .60) = 0.0 / .40 = 0
In other words, when one rater's ratings have no variation, the agreement corrected for chance (kappa) is considered to be zero; whereas the raw agreement (in this instance) is .60.
Good luck with your work!
Thanks David! So what would you actually report in a study? The raw agreement?
If I understand your question properly, have you considered McNemar's test of change which deals with categorical data for participants who may change category membership at two testing points? There is a link here with a calculator and it will deal with zero. It will give you CI for odds ratios which you could then compare with the ones you get in other analyses. Results could also be compared to effect size benchmarks perhaps.
http://scistatcalc.blogspot.com/search?q=mcnemar
Thanks everyone! So I get the impression that the McNemar test is still superior to percentage of matching responses?
Percentages are helpful and illustrative but McNemar will give you CIs and the potential to report effect sizes which seems more rigorous to me. It all depends on your audience.
Hello Jeremy,
Proportion agreement is a simple statistic, unlikely to be misunderstood. McNemar's test evaluates whether there is a significant shift in ratings from one to the other (judge/rater/source). While agreement and McNemar's tend to be inversely related, it's not a one-to-one correspondence. You could report both (though, with empty off-diagonal cells, McNemar's test won't be powerful); but it sounded to me as if your concern was that of concordance across raters, not change/shift.
Good luck with your work!
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS