This is the first time that I have tried to do this within qualitative research. The process seems quite difficult with regards to the different coding styles of qualitative researchers. I'd love to know people's thoughts, or any research people have read describing the process. Thank you in advance.
I think a lot depends on what your purposes are and where you hope to publish. For example, if you are doing a qualitatively oriented content analysis and have counted your codes, then a score on inter-rated reliability would be expected. Alternatively, If you have done a largely interpretive analysis and plan to go to a qualitatively oriented journal, it would add little.
Kappa reduces agreement and disagreement to a single number. I think that the examination of disagreements is more informative. In some simple cases, one rater misses something that the other spotted. In a more significant case, one rater thinks something is there and the other does not. And in the really interesting case, both raters think there's something there, but they don't agree on an interpretation.
Examination of disagreements should be part of the process, rather than an analysis post facto, I believe.
Cohen's kappa coefficient measures the agreement between two observers in their respective classifications. We have used it in NVIVO to contrast the concordance between researchers. It offers a complementary measure. High concordance provides reliability to encodings. A very low concordance reports on different approaches in researchers. We have also used it to check concordance in Delphi panel studies.
- Badges
- Science method
- Similar topics
- Methodology
- Research Methodology
- Qualitative Research