Dear peers,
I am currently working on a very large qualitative project. We have 46 interviews and a very complex coding scheme: 2 topics, 6 sub-topics each with each sub-topic having about 3-4 further categories with sub-categories.
At the moment, I am having a hard time finding an optimal way to report the agreement between the two researchers who iteratively code the material. The Perreault and Leigh measure, Krippendorff's Alpha, or Cohen's Kappa all seem to work for small coding schemes, but I really struggle in terms of applying them to my project.
Personally, I think that - as we apply Grounded Theory - an open discussion and joint development of the coding scheme and coding paradigm would be a good way to establish agreement, but that is just my perception. In your opinion, what would be the best way?
Every piece of advice is highly appreciated. Thank you very much!
Yours sincerely,
Franziska
Krippendorff's alpha is currently considered to be the state of the art for inter-rater reliability.
How about the Intra-class Correlation Coefficient (ICC)? By the way, as I know, on-going discussions to acheive the agreement with your colleague is essential in qualitative research. Use of statistical methods are not prohibited; however, um... I think that it would be better to find references related to that.
Dear F,
my co-author and I were in the exact same situation half a year ago... it is really frustrating. We use inductive content analysis and applying the suggestions of Elo and Kyngäs (2008) really made us move forward quite “easily” althogh it took us many, many hours to discuss, agree and make explicit WHY one word/phrade/verbal behavior should be coded as “x” and not as “y” or “z” etc.
The development of clear definitions to each code, sub-group and generic category made it much easier for us to code - and for one of our doctoral students to “test code” ten pages of transcription and compare her coding with our coding. In this process we ended up revising some of the definitions to the codes and to the sub-categories. This is just one of the steps we applied to mitigate the interrater issue. If you use Nvivo the system has some excellent functions to enable and test this work.
best,
Jacob
Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of advanced nursing, 62(1), 107-115.
Thank you very much for your helpful responses. Your comments are highly appreciated.
I have seen papers published that do not report any kind of inter rater agreement statistics. How do you feel about that?
Best regards,
Franziska
To my experience and knowledge, using well-known inter-rater agreements calculations do not seem to fit the situation when one choses to apply Grounded Theory approach. As I have understood from your question that you have already started to code in an iterative way then the following article may be of great help.
Campbell, J. L., Quincy, C., Osserman, J., & Pedersen, O. K. (2013). Coding In-depth Semistructured Interviews: Problems of Unitization and Intercoder Reliability and Agreement. Sociological Methods and Research, 42(3), 294–320. http://doi.org/10.1177/0049124113500475
The most common use of inter-rater reliability is when you have a pre-determined coding system and you want to ensure that each coder is using it an identical fashion. This situation fits well with deductive coding systems in content analysis.
It is not usually a good fit in Grounded Theory, however, where the coding system emerges inductively during the analysis, and typically evolves throughout the coding process. In that case, "an open discussion and joint development of the coding scheme and coding" should occur throughout the coding process.
However, it sounds as if you have completed the data collection, prior to analysis, which is not the preferred way of doing things in GT. Further, it sounds as if you want to develop the coding system prior to applying it to the data as whole, which is even further from what is advocated in GT.
Can you say more about why you want to use Grounded Theory?
- Badges
- Science method
- Similar topics
- Methodology
- Research Methodology
- Qualitative Research