We want to measure the level of agreement between several raters. They classified drug related problems for 20 different scenarios. We used an adapted PCNE classification where they coded Problems, Causes and Interventions. Since the data are nominal we aimed to use the Fleiss Kappa. However, the categories are not mutually exclusive (e.g. there are several possible problems in one scenario) and the raters used several different codes in Problems, Causes and Interventions section. It seems that the Fleiss Kappa needs mutually exclusive categories. Any ideas on how to measure the level of agreement in such cases?
This study has used cohen's kappa may be that could be of some help.. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3605495/
You may apply scoring method on your output, thereafter you may take decision based on their score.
Hi Nejc Horvat
I am leading a research about calculating the inter-annotators agreement without mutually exclusive categories. I read a lot and there are different of methods, I feel a little bit confused. Any recommendation or help will be really appreciated.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA