I would like to calculate the Inter-Annotator Agreement (IAA) for a dataset including entities and relations. The dataset has been labelled by several annotators and the labels itself have a frequency. I am getting a bit lost about the possibilities to calculate the agreement.
First of all, Scott's Pi (or Siegel & Castellan's Kappa) appears to be better than Cohen's Kappa dealing with bias (see for instance here: https://www.aclweb.org/anthology/J04-1005.pdf). In more recent related work I have seen the recommendation to use the F-Score instead, dealing with entities (see for instance here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3540456/pdf/amia_2012_symp_0144.pdf). To calculate the agreement for relations, I would probably also apply F-Score.
If you disagree, you are welcome to give me suggestions here. However, the questions I actually have are the following ones:
1) If I have more than two annotators, how would I ideally put the agreement into the F-Score? Shall I do that pairwise? And if labels are unbalanced micro F-Score for each annotator and the then macro over all annotators?
2) The annotated relations suffer from falsely annotated entities. Therefore if I just calculate the F-Score, I might "punish" an annotator twice, if the person did not label the related entities in the first place. So the annotator would be not able to annotate the relation at all. Therefore would it make sense to consider only those relations where the same entities are given? Any suggestions?
Thank you!
I elaborated on questions of the IAA in my thesis on opinion mining and lexical affecting (Section 9.3, dialogue corpus annotated by four persons). https://www.researchgate.net/publication/278383087_Opinion_Mining_and_Lexical_Affect_Sensing. For example, the final IAA was considered as the most frequent annotation.
HiH, Alexander
Dear Alexander, thank you for your reply! Unfortunately I am not sure, whether I fully understand. Section 9.3 is part of the conclusion. Is this the part you would like to point out to me? And, does "For example, the final IAA was considered as the most frequent annotation." refer to my first question? Do you suggest to focus on the more frequent annotations (labels)? Best, Roland
For entities we use parallel F-Scores and the code handling BRAT annotations is here: https://github.com/kldtz/bratiaa
Dear Christina, thank you for pointing this out! As we use BRAT, this might be very helpful :-)
- Badges
- Science topic
- Similar topics
- Filing