Currently working on a meta-analysis article and had 2 raters rate the quality of the articles obtained from review for inclusion. Now I would like to analyze the level of agreement between these two raters. Kappa would take a great deal of transcribing so I decided to use ICC (this way I can just use total scores for each scale from the measure). Though I know 0.7-0.8 is acceptable, I cannot find a resource to cite. I'm also not certain if the sig. figure SPSS provides should be used for my purposes. Thanks so much in advance!
The significance of an inter-rater reliability metric such as the ICC is usually meaningless. It doesn't matter that the score is greater than 0, we want to know if it is greater than some heuristically derived threshold that is normally much larger than 0. There is value in reporting its confidence interval, however, as well as in calculating the significance of a comparison with a specific threshold value (e.g., 0.75).
To answer your question about citations for ICC heuristics, many people cite the attached paper by Cicchetti (1994) and the following rules. It should be noted, however, that many people find such rules to be oversimplified and argue that the "bar of excellence" should depend on the application and literature involved.
Less than 0.40—poor.
Between 0.40 and 0.59—Fair.
Between 0.60 and 0.74—Good.
Between 0.75 and 1.00—Excellent.
Article Guidelines, Criteria, and Rules of Thumb for Evaluating Norm...
I always use Portney and Watkins scale (2009) is slightly more rigorous than that one mention above. You can try both and make a choice which would be more appropriate for your data:
https://www.pearsonhighered.com/program/Portney-Foundations-of-Clinical-Research-Applications-to-Practice-3rd-Edition/PGM274308.html
In one of the chapter you will find a a scale.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS