Hello folks. I am currently carrying out Qualitative Content Analysis on Interview data, 7 questions total. I've gone through all my transcripts and generated a big old list of basic inductive categories but before I actually come up with a formal coding scheme, I thought it would be best to look into methods of reliability and validity to ease the whole coding process.
I've had a look at Krippendorff's Alpha, which seems to be a winner, and I'm slowly figuring it out but I'm having some difficulties visualising how my eventual coding system is going to fit into what needs to be done. I understand that for these methods you generally need to compare each Judge/Rater's unique coding for a specific case (i.e. in this case, an interview participant) and use this information to determine the level of agreement between all raters.
However, in my system each case/participant is going to have multiple codes (i.e. derived from each answer to the questions) that are likely to occur multiple times within one case (i.e. Recurring Themes for Different Questions), which seems, at least to me, to be incongruous with how Krippendorff's Alpha works.
Am I looking at the problem wrong or am I missing something?
Cheers
Paul
Yes, there is a lot to be said for using Krippendorff's alpha, but I think you need to pay more attention to the units of analysis for your coding.
In particular, the traditional approach is to "unitize" the data. so that there are unique segments of data where a code either is or is not assigned. Setting up each interview transcript as your unit is so broad that it will indeed create the kinds of problems you described. At the other end of the scale, dividing transcripts into paragraphs often requires a good deal of subjective judgement -- but in my experiences, this is the most common way that people set up a assessment of inter-rater reliability.
Krippendorff’s alpha (Kalpha) is a good choice to evaluate the quality of inter-rater reliability. Hayes and Krippendorff (2007) said that a Kalpha of 1.000 represents perfect reliability whereas Kalpha = .80 is often thought of as the norm for good reliability. A minimum of .67 or even .60 may still be satisfactory, if the coding is complex, such as the somewhat subjective determinations that were required of raters in this study (de Swert, 2012).
With Kalpha, you compare each data point between multiple coders, and then Kalpha tells you how similar the coding is, and thus the intercoder reliability.
I think following David's answer here gives you a specific response and path to follow, I would use this approach.
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology