In my data, there are roughly 400 different patients, each of whom was rated on one continuous scaled test by two different raters. The trouble is, there are many different raters. So, I have two different scores per patient, many the same, some slightly different-how could I characterize the rate of agreement in the scores overall? Also, any thoughts on setting up the data frame for analysis would be helpful. Jerry
HI. You can calculate Cohen’s Kappa for score given by the two raters, If Kappa is higher than 0,75 you are ok. When you find a question for which Kappa is lower than 0.75 than the two raters have to discuss their scores until they find a common evaluation. When the Kappa is 0,75 or more for each questions you are ok, and you can use the scores given by one of the two raters as you prefer.
I guess, Jerry, you challenge is real. Kappa is OK when TWO raters are the same across your dataset. Technically, Kendall's coefficient of concordance W would be the alternative when ALL the cases are judged By MORE than two judges. Your case seems to be somewhere inbetween Cohen and Kendall. Both statistics would be a bit biased, I guess. You could approximate the p-value in-between those approaches? Another option would be to turn to non-parametric IRTmodeling. Within that tradition, there are procedures where the behavior of the scorers are modelled. I cannot lead you much further in that direction, though. Check this kind of articles: http://www.columbia.edu/~ld208/jem05.pdf
You can set a criteria to select the raters. Also, giving weight (based on raters experience, qualification, education etc)to each raters might help to reducing the human effect.
In this case, determining the Intraclass Correlation Coefficient would do good for you. Inter-rater reliability is evaluated by examining the scores of two or more raters given independently and calculating the correlation coefficient between the two sets of the scores using intraclass correlation coefficient in SPSS. The results of the correlation coefficient would tell the consistency of the scores given by the raters (Sak, 2008). Inter-rater reliability measures the consistency between differing raters or examiners which is the degree of correlation between two or more raters for the purpose of determining whether or not they score respondents’ responses similarly. Typically, any value above 0.70 is acceptable, above 0.80 is good and above 0.90 is excellent. In case you are not familiar how to run Intraclass Correlation Coefficient in SPSS, you can refer to the following link to help you do the job. Hope you find it useful.
https://www.youtube.com/watch?v=v2d6eyYoh4M&t=322s
Regards,
Karwan
- Badges
- Science topic