I'm calculating an ICC for inter-rater reliability of a measurement on a radiograph. Rater 1 read the radiographs 3 times blind to the other reads to calculate intra-rater reliability. Now raters 2 and 3 have read each radiograph 1, and we need to calculate inter-rater reliability. Should I calculate ICC with the average of rater 1's measurements and measurements of 2 and 3 for three separate measurements? Or with all 3 of rater 1's measurements and the 2 others, for a total of 5?
Hi,
On the premise that IRR is a way of quantifying the degree of agreement between two or more coders who make independent ratings about the features of a case, a set of subjects, etc, ideally you should calculate the coefficient (Cohen’s kappa (nominative v) OR intra-class correlations (ordinal, ratio,etc v)) of the ratings the three raters assigned to the same radiograph(s) and then average.
However, you find the paper by Hallgren's also useful as it offers the approaches to calculate the IRR for ill-structured measurement design:Article Computing Inter-Rater Reliability for Observational Data: An...
- Badges
- Science topic
- Similar topics
- Social Science
- Sociology