I am grading quality of studies with a questionnaire which has 20 questions. Scores vary from 0 to 1 for all but two for which score varies from 0 to 2. I need to evaluate 50 studies. This will be done by 2 reviewers. To find the test retest reliability or inter rater reliability, should I add total score of 20 questions for each study, So I have 50 values from reviewers 1 and 50 values from reviewer 2? How can I find correlation between individual scores of 20 questions of each study?
For the 0/1 questions, use the kappa statistic. For 0/2 use weighted kappa. For a summative score, use the intraclass correlation coefficient.
Hello Dnyanesh,
I would like to help, but I am not clear about the nature of your data. If you have 20 items on 50 studies, then the first rater will have 20 x 50 individual pieces of data (1,000 pieces) that you want to compare with the corresponding 1,000 pieces of data from your second reviewer. That would mean 1,000 kappa analyses. I can't imagine that this is really what you want to do.
Instead, and I could be wrong, I suspect that each set of 20 items measures the same thing, and that you want to see if the total of those 20 items, which you had each of your two raters obtain on 50 different people or situations. Is that the case? If so, I would support Adrian's suggestion above, namely that you use the intraclass correlation coefficient (ICC). That would show you how much agreement there is between the two raters across 50 paired items of data. I can give you a bit of advice regarding ICCs if you like, but essentially you would obtain a single statistic that would reveal the amount of agreement between your two raters.
Robt.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS