I am aiming to look at interrater reliability.
The exact tests are yet to be decided, but I'm working on the assumption that the resultant data will be ordinal (or can be coded as such).
I want to look at the interrater reliability between two groups (expert and novice).
I believe I'll be using the krippendorff alpha based on the flexibly for missing data, better for multiple raters....and it's what others have done!!
For my proposal they want a sample size estimation. I can find examples for Cohen's Kappa and the usual power/sample size calculation, but I'm not sure which to use.
Also, would I be needing the sample size as per raters or data points. Ie 4 raters scoring 100 test would give more data points than 100 raters scoring 3 tests.
Any signposting or comments would be great. Thanks
Perhaps not a direct answer, but Krippendorff works fine in practice.
Source: Calculating inter-coder reliability in media content analysis using Krippendorff’s Alpha
Knut De Swert (University of Amsterdam) – First version (01/02/2012). Please send your comments, suggestions, additions and corrections to [email protected] To quote: Sample size, multiple (more than 2) coders or missing data are not problematic for calculating KALPHA, and all measurement levels can be tested.
Re-found Krippendorff' table of sample sizes:
University of Pennsylvania Scholarly Commons Departmental Papers (ASC) Annenberg School for Communication 2011
Agreement and Information in the Reliability of Coding
Klaus Krippendorff University of Pennsylvania,
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size