Hi!
I'm doing interrater agreement analysis and want to do Cohens' kappa on all combinations of 14 raters. This is the equivalent of "if 14 guests at a party all shake hands, how many handshakes will there be". For 14 participants that is 91 unique pairs.
The syntax for analysing one pair is straight forward;
CROSSTABS
/TABLES=rater_1 BY rater_2
/FORMAT=AVALUE TABLES
/STATISTICS=KAPPA
/CELLS=COUNT
/COUNT ROUND CELL.
Then changing the first rater
CROSSTABS
/TABLES=rater_2 BY rater_3
/FORMAT=AVALUE TABLES
/STATISTICS=KAPPA
/CELLS=COUNT
/COUNT ROUND CELL.
But how do I use syntax to repeat this, changing to rater_3, rater_4, rater_5 etc., which would be easier than copying the CROSSTABS-command 91 times? The DO REPEAT-command can loop an analysis but I haven't got it to change one variable at a time by each cycle.
Hi Rasmus Walter Green, Nope, there is no such facility available in SPSS. I too faced this problem. Only option is to copy syntax 91 time. :(
Dear Rasmus,
first, as far as I know, Cohen's kappa isn't the most useful choice when comparing more than two raters, because of - as you implicitly mentioned yourself - n above k resulting combinations. Furthermore, to my limited knowledge, SPSS cannot perform an automtated Cohen's kappa analysis for n=14 raters.
Maybe you want to use other statistical measurements such as e.g. Fleiss' kappa?
Hope this could be of help.
Best regards
Florian
No, you are absolutely right. When comparing multiple raters Fleiss' kappa or ICC is the way to go. I did that but now I want to investigate if there is poorer agreement rater vs. rater in raters with low accuracy (determined by their assessment compared to the gold standard).
So first I've checked each rater and calculated each sensitivity, specificity and accuracy in detecting my disease. Then I wanna see if highly accurate raters have higher interrater agreement than poorly accurate raters.
This might lead to the conclusion that my diagnostic method is good, as long as it is in competent hands, but let someone inexperienced use it and you will get uninterpretable results with low diagnostic accuracy and low interrater agreement.
Hi Rasmus Walter Green, Nope, there is no such facility available in SPSS. I too faced this problem. Only option is to copy syntax 91 time. :(
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS