I'm doing a trial where I have to calculate intra- and interrater agreement with Cohen's kappa. There are 412 measurements/examinations with ultrasound and for each, there are seven binary yes/no questions for rater 1 and rater 2. How many of the 412 do I have to use to calculate Cohen's kappa?
Why does sample size matter here? You're interested in how correlated the ratings are so I would use all of them. David Booth
It is very time demanding to look at all 412 scans. The intra-/intervariability calculation is only done as quality assurance. It is "tradition" in the my field to do it on a subsample of 10-30% of the total sample but this is not based on a statistical calculation. So my question is whether it is possible to make a calculation of how big the subsample has to be. I assume that the statistical power would not increase beyond a certain number.
I found this article on the subject which states it is possible to make a sample size calculation: Article Guidelines of the minimum sample size requirements for Cohen’s Kappa
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude