In my research I use thematic content analysis to describe legal documents. After building our coding framework, we have assigned a sample of the data between 3 independent raters. I have asked them to code the data and upon receiving their codes back, I calculated the score by dividing the total number of codes present with the matching score. The simple percentage score between the 3 raters is 88%. I tried to calculate Krippendorff's Alpha using Recal3 (online), Real statistics (Add-in Excel) and with a R Script (icr package in order to calculate Krippendorff's alpha).
I always get aberrant data : -0,46 or -0,26 or a really low Kalpha like 0,10 or 0,11.
Why do you think this happens even if I have 88% of agreement?
My tables are composed of 3 rows (corresponding to the raters) and 26 columns (corresponding to the codes). The data is nominal, it only reflects the presence (1) or absence (0) of the category for the result of each encoder. The table is in the attached file.
Thank you
Hello Saray,
The most likely reason is that Krippendorff's alpha, like Cohen's kappa, relates the degree of agreement beyond that expected by chance. So, even though the raw agreement might be high (you mentioned that 88% of paired judgments agreed), apparently this is not very different from what could be predicted by the marginal rates across raters and cases. Hence, the low estimated values for alpha.
Good luck with your work.
Hi David Morse
Some newspapers ask for this coefficient in order to publish, but I will take this into account. Furthermore, there is research that addresses the issue of the "paradox" of alpha, to be taken into account in these circumstances.
Thank you very much
I personally think that very high rates of agreement are all that really matters. As David Morse notes, various coefficients of agreement can be highly sensitive to marginals. Unfortunately, too many reviewers blindly demand kappa or alpha, despite near perfect agreement among coders.
Hi, David L Morgan
I have been "blindly" looking for answers in traditional coefficients for some time. Clearly, even if the coefficients do not agree, our work can incorporate a thorough process of inter-coder agreement. Reading such responses from experienced researchers makes me confident.
Thanks for your reply.
Saray Chavez I totally understand the frustration. I see that you have binary ratings which is why your "agreement by chance" is high (well technically Krippendorff's alpha measures disagreement by chance which will be low). That's already one reason for the low alpha value, but also I can see that "1" is chosen much more often than "0." This means that the categories are NOT equally probable, and that adds to the lower "agreement" as measured by alpha. Actually, you can have the exact same percentage of the agreement but very different alpha or kappa just because of these issues...
Personally, I think that raw agreement should at the very least be presented with the alpha value, and really the alpha value is better at distinguishing between two populations of raters rather than showing high or low agreement.
EDIT: Actually, I've just checked your data and now your raw agreement is about 81.5% (raw between all 3 raters). The raters agree on 22 cases and disagree on 5. All cases they agree on are annotated as "1" meaning that "1" is more probable answer (and that's the reason for low alpha). If you change 11 of these cases to an agreement but with label "0" your alpha goes up to 0.75! (from initial 0.10). The raw agreement stays, of course, the same... I like to call it alpha magic though I wish more reviewers understood exactly what these numbers mean.
Also, once I change the number of labels from 2 to 5 (again keeping the same number of raw agreement) alpha jumps to 0.83. This is possibly more reasonable, as having the same high agreement on much more categories is a "stronger agreement" (less possible by chance).
Hi Marzena Karpinska
First of all, thank you for you're answer and for taking time to explain this.
I have read it twice to understand the logic of what you explain about K alpha, honestly it is difficult to find such explanations in theoretical texts.
So according to what you tell me the K alpha helps to measure disagreement by chance between coders rather than agreement.
To do the test, did you change 11 boxes randomly or is there an explanation behind that?
Do you know how I can justify the change of labels from 2 to 5 in a methodological way?
Regards,
Saray
Hi Saray Chavez
No problem, first of all let me clarify something. I did not suggest you look at the disagreement by chance, Krippendorff's alpha does that (as opposed to Cohen's kappa which calculates the agreement by chance). What I meant was that the expected agreement or disagreement BY CHANCE, which is used to calculate kappa and alpha respectively, depends also on whether all categories are equally probable.
In your case, all the agreement is on the label "1" and that affects your results. You cannot do anything with this but just to illustrate my point I randomly changed 11 items with an agreement on label "1" to an agreement on label "0" leaving the actual raw agreement the same. You can try it yourself, and you will see how that value will change. Sadly, this is only to illustrate why you have the value you obtained. There is no way to increase that value, and normally I think you would just report the raw agreement and alpha hoping for reviewers to understand.
As for changing for 5 labels, I did not suggest that, just illustrated that the value of alpha will be higher the more labels you get (while keeping the raw agreement constant). This is understandable, as you would expect less agreement by chance on more values.
So to sum it up, I don't think you can do more than just trying to understand where that value comes from and also that low alpha is not necessarily an indication of a low agreement.
If you are looking for anything to cite, here is one paper suggesting to report also raw agreement: https://aclanthology.org/W19-8643.pdf
Marzena Karpinska I can clearly see that you are not suggesting but explaining the phenomenon of Krippendorff alpha.
This therefore reinforces the answers given by other researchers on the calculation of K alpha. It has helped me to know what's behind coefficents and to reinforce the idea that it is not necessarily wrong to simply include the raw agreement.
Thank you !
A bit too late, but this may be useful for anyone struggling with the same issue.
As it was mentioned before, this is not very surprising, as both the expected agreement and the actual agreement come from the same distribution (many items with the same label => expect many of these labels to be there => need even more agreement for high coefficient value).
Here is a good resource to cite and also look into:
Book Statistical Methods for Annotation Analysis
For this particular issue, please see section 2.5.2 in particular.
Thanks for your answer Marzena Karpinska I will look the resource you shared.
Thank you
You may find that the following two articles have explained the problem you encountered.
Article Mistakes and How to Avoid Mistakes in Using Intercoder Relia...
Article Do Not Force Agreement: A Response to Krippendorff
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Graphs