Hello!
I made a small post mortem computed tomography study on 5 corpses with 5 different scan protocols to evaluate the "best" protocol (=best image quality) for a native cardiac examination. Therefore i created a questionnaire with 8 questions (for every corpse) and a 5-scale rating (very good; good; acceptable; bad; very bad). I have at least 4 raters, so I need a interrater-reliability for multiple raters and I also have many ratings (1 observer gives 200 ratings)
There are a few options, but I don't know which one I should use: Fleiss Kappa, Krippendorffs Alpha, Intra-class-Correlation, Kendalls W
At first I tried the Fleiss Kappa, but the value was very low (0.12) - The reason for that is probably that I have to calculate the weighted Fleiss kappa, but many scientific papers say that Fleiss Kappa is for nominal data.
So I´m a bit confused right know, because I think there's not really a consensus which interrater-reliability is the "best".
I hope someone has more experience and could give me an advice for my study.
Best regards,
David Riegler
Kappa seems most appropriate to me, the exact agreement was probably just that poor. Re-group your variables where you calculate the Kappa value with only 3 categories: 1. very good/good 2. acceptable 3. bad/very bad. Hopefully, this will improve your agreement to at least fair. Report the 0.12 data for exact matches and the new value for 3 categories. Try to report the 0.12 data somewhat in passing, e.g., when accounting for exact matches, there was no agreement (Kappa=0.12), however, when grouping [...] agreement was slight/fair/good agreement. I'd prob be best to use the slight/fair/good scale instead of the minimal/weak/moderate scale (same values different descriptors).
Consider having two or three of your study raters review it again in consensus and use that as another (maybe the main study metric). If that's not feasible, then try to find either exact matches or 'exact' match when pairing down the variables to 3 categories instead of 5. It doesn't look like the bottom line of your study is going to be excellent agreement among the five study readers.
- Badges
- Science topic