Hi Rasmus Walter Green, Nope, there is no such facility available in SPSS. I too faced this problem. Only option is to copy syntax 91 time. :(

Dear Rasmus,

first, as far as I know, Cohen's kappa isn't the most useful choice when comparing more than two raters, because of - as you implicitly mentioned yourself -   n above k resulting combinations. Furthermore, to my limited knowledge, SPSS cannot perform an automtated Cohen's kappa analysis for n=14 raters.

Maybe you want to use other statistical measurements such as e.g. Fleiss' kappa?

Hope this could be of help.

Best regards

Florian

No, you are absolutely right. When comparing multiple raters Fleiss' kappa or ICC is the way to go. I did that but now I want to investigate if there is poorer agreement rater vs. rater in raters with low accuracy (determined by their assessment compared to the gold standard).

So first I've checked each rater and calculated each sensitivity, specificity and accuracy in detecting my disease. Then I wanna see if highly accurate raters have higher interrater agreement than poorly accurate raters.

This might lead to the conclusion that my diagnostic method is good, as long as it is in competent hands, but let someone inexperienced use it and you will get uninterpretable results with low diagnostic accuracy and low interrater agreement.

Hi Rasmus Walter Green, Nope, there is no such facility available in SPSS. I too faced this problem. Only option is to copy syntax 91 time. :(

Adil: thank you, well now I at least know not to look for a solution that is not available. I actually added some more raters, so now I have 105 pairs!