In my research I use qualitative content analysis to describe my interview data. After building my coding framework, I have assigned a sample of the data between 2 independent raters. I have asked them to code the data and upon receiving their codes back, I calculated the score by dividing the total number of categories present with the matching score. The simple percentage score between the 2 coders is 75,3% which would be fairly good. Do I need to calculate Cohen's Kappa coefficient as well or would the percentage agreement be enough for a publication?
I think it depends on the field in which you are working and the standards of the journals to which you would be submitting. For some kinds of qualitative work (especially where you are relying on subjective interpretation), there is very little interest in inter-coder reliability. In other fields (such as communication and media studies), a simple percentage of agreement would not be enough.
So, take a look at the journals you are considering and determine the standards that they use.
How many categories are involved? If it's a dichotomous decision, then you absolutely must report the kappa coefficient, and the result may be a little discouraging (especially if the cases aren't split pretty close to 50:50). If the judges are choosing from among (say) a dozen different categories, and agreeing 75% of the time as to exactly which category the case belongs to, then your results are pretty impressive, though you'd be on firmer ground if you also reported kappa (or maybe Krippendorf's alpha). As you probably know, the whole point of these procedures is to correct for the level of agreement that would be expected if the judges were acting randomly.
I think the best response, though, is to ask you a question in return: Why not go ahead and calculate the higher-level statistic? It isn't as though this will require you to master a new skill set or anything; calculators for Cohen's kappa (and even Krippendorf's alpha) can be found on the web & used without charge. They're quite easy to use, too!
I would suggest, Cohen's Kappa due to simple reason that it also takes care of chance agreement and hence to that extent will represent true agreement. However it still does not cover the 'category mismatch' i.e. both raters agreeing on a common category but that is not the intended category. For which you might think of using Moore & Benbasat Hit ratio.
Hope this helps.
Fully agree with Stephen on the use of higher-level statistic. Here is a citation to a paper that covers a lot of issues related to IC reliability.
Lombard, M., Snyder‐Duch, J., & Bracken, C. C. (2002). Content analysis in mass communication: Assessment and reporting of intercoder reliability. Human communication research, 28(4), 587-604.
I am doing a similar method in my study where I used content analysis and coding. I was planning to use Cohen's kappa but the statistician advised to use a percent of agreement instead because of the small sample of data. I am measuring the inter-rater reliability for only 25% of my data to assess any drift in rating. It varies according to the sample number.
I am curious which stats did you use for your study?
Use of kappa is being discouraged as a survey of the Kappa literature would show. One major handicap referred to as Kappas's paradox indicates that with more agreement there is a contradictory decrease in Kappa values which can confuse the non statistician as it is very much counterintuitive.
Amanuel Gebru are you able to share a link to the survey you mention? Many thanks.
I refer you to the below . You may further find a methodologically superior alternative in the guide to ATLAS.ti
Measuring Inter-coder Agreement – Why Cohen’s Kappa is not a good choice- (suggesting you may try Krippendorff 's alternative).
Why Cohen’s Kappa should be avoided as performance measure in classification
- Rosario Delgado ,
- Xavier-Andoni Tibau
📷
- Published: September 26, 2019
- https://doi.org/10.1371/journal.pone.0222916
- Badges
- Science topic
- Similar topics
- Biological Science
- Bioecology
- Systems Ecology
- Landscape Ecology
- Connectivity