I'm searching for a measure by which I'm able to check if one coder gave consistent answers while coding (with reference to determinated categories) or if there is/are any bias or effects (e.g. he/she could differ/change in his/ her assessment) . Does anybody know something like this or a helpful reference?
There are a few different ways to assess credibility. Some would say that as its a qualitative method you shouldn't try to quantify agreement - especially if your qualitative analysis subscribes to a social constructionist perspective.
There is one paper that I have referred to in the past where you actually calculate Cohens kappa to check for reliability. Extracts from the transcript are divided between coders, and you check whether they assign them to the same theme/cluster or not. Its described in this paper: http://www.tandfonline.com/doi/abs/10.1080/10503307.2011.565487
You could also include a reflexive statement - where the code reflects on their biases and how this might effect their coding. A lot of papers just say that the themes/clusters were shared with the wider research team to check for credibility.
Good luck!
I think that any analysis of agreement between raters should involve discussion of differences. This may result in a richer, clearer coding scheme. There is a tendency to view agreement analysis as something that is done as an exercise in itself rather than as a way of refining the analytic approach.
- Badges
- Science topic
- Similar topics
- Psychology
- Psychometrics