Hi everyone,
For my studies on the IRR I have binary data and three raters. But on one variable they all score a 0 for all subjects. This means that everything is rated as a 0. But SPSS then gives me the warning that every outcome is the same and that the command stopped. What do I do with it and what do I have to report in my studies?
Hopefully you can help me!
I would delete that item from the kappa calculation, and then when you report your coefficient, note that one item was excluded due to 100% agreement.
But weirdly I do have items that have a kappa of 1 because of its full agreement. But in this case it is not because there is no variation in the given scores. I read somewhere that that is the case. And I suppose I will report at as you say David.
You can have full agreement when the target variable varies, so long as each rater gives it the same rating.
What you have, however, is complete agreement that the target scores zero, but no information on how raters would have responded if the target showed any variation. So the design is not sufficient to assess agreement across the range of possible target values. Does that make sense?
That makes sense, thanks! But I don’t know how to change it. Because this is regarding a test for narrative development and it is possible to score zero’s all over. in this case I am also testing the individual items on their agreement to see if there are some items we have to adress. Is there something else I need to do in this case or just report it?
- Badges
- Science topic
- Similar topics
- Physical Sciences
- Materials Science
- Materials
- Inorganic Materials
- Metals
- Alloys