In my data, there are roughly 400 different patients, each of whom was rated on one continuous scaled test by two different raters. The trouble is, there are many different raters. So, I have two different scores per patient, many the same, some slightly different-how could I characterize the rate of agreement in the scores overall? Also, any thoughts on setting up the data frame for analysis would be helpful. Jerry
HI. You can calculate Cohen’s Kappa for score given by the two raters, If Kappa is higher than 0,75 you are ok. When you find a question for which Kappa is lower than 0.75 than the two raters have to discuss their scores until they find a common evaluation. When the Kappa is 0,75 or more for each questions you are ok, and you can use the scores given by one of the two raters as you prefer.
I guess, Jerry, you challenge is real. Kappa is OK when TWO raters are the same across your dataset. Technically, Kendall's coefficient of concordance W would be the alternative when ALL the cases are judged By MORE than two judges. Your case seems to be somewhere inbetween Cohen and Kendall. Both statistics would be a bit biased, I guess. You could approximate the p-value in-between those approaches? Another option would be to turn to non-parametric IRTmodeling. Within that tradition, there are procedures where the behavior of the scorers are modelled. I cannot lead you much further in that direction, though. Check this kind of articles: http://www.columbia.edu/~ld208/jem05.pdf
You can set a criteria to select the raters. Also, giving weight (based on raters experience, qualification, education etc)to each raters might help to reducing the human effect.
- Badges
- Science topic