I did my pilot study through semi structured interview and I used thematic analysis for coding. Now, I want to evaluate the reliability of my coding. I read about the ways of evaluating the validity in thematic analysis such as inter-rater reliability. I found that this method needs two independent coders. Could you recommend another way that I can do individually?
Due to the subjective nature of qualitative research, it is incumbent upon the researcher to apply several strategies to ensure that the information obtained truly captures the voice of the population of interest (Lietz, Langer & Furman, 2010). The term “rigor” in qualitative research is analogous to achieving ‘validity’ in quantitative circles. According to Lietz, et.al. (2010), rigor suggests that the research has been subjected to efforts that increase confidence that findings are representative of the meanings presented by our participants. These efforts should place awareness on how we interpret data in order to minimize reactivity, bias and in essence place priority on the voices of the participants. The term 'trustworthiness' is utilized in qual research and that is the goal to achieve, so, if you are not part of a team and are the only rater - then you need to have a variety of rigorous initiatives in your design directed towards establishing trustworthiness. As already mentioned, member checking (Charmaz, 2011) is one way of doing this as well as establishing a paper trail (Charmaz, 1990), being cognizant of your 'positionality' and being reflexive in the process. A good qualitative research text such as by Patton (2002) will illustrate these methods of establishing trustworthiness in great detail.
Dear researcher,
Coefficients that assess the reliability of data making processes – coding text, transcribing interviews, or categorizing observations into analyzable terms – are mostly conceptualized in terms of the agreement a set of coders, observers, judges, or measuring instruments exhibit. When variation is low, reliability coefficients reveal their dependency on an often neglected phenomenon, the amount of information that reliability data provide about the reliability of the coding process or the data it generates.
To get more information please read this artical:
http://repository.upenn.edu/cgi/viewcontent.cgi?article=1286&context=asc_papers
It depends on the type of reliability you are looking to evaluate. An independent coder would be used to check inter-rater reliability. For "conceptual" reliability (to avoid having your own thoughts projected into your analysis 100%), you could also use member-checking. This would mean that you contact a percentage of the participants (preferably one out of three) and see if your analysis of the interview they participated in also represents their perception of the content. This also raises the ethical standards you apply in your qualitative research.
This is an interesting question. The concept of reliability inherently suggests comparison. Working as a team clearly enhances this, though I think concepts like 'inter-rater' reliability are reliant on a fixed template or coding scheme which might not be the case in Thematic Analysis. Also the OP has requested ways of assessing reliability that do not rely on another researcher.
So the options that are left (logically), are some pre-existing standard, another person that is available (eg research participant), or the reader of the research. The first can be addressed by comparing against existing knowledge (which will be done anyway as part of the research process) but has the downside that it tends towards priveleging replication. The second can be done by member checking, but this relies on the participant having the same level of knowledge and analytical skill (unless you are only checking data which has its own issues - re-narration; social desirability etc).
The third option orients the researcher much more towards documenting the analysis process such that it becomes more transparent to the reader. This seems the most fruitful route in more interpretive work. In this respect, I recommend the following:
Angen, M. J. (2000) ‘Evaluating interpretive inquiry: Reviewing the validity debate and opening the dialogue’, Qualitative Health Research, 10(3), pp. 378–395.
Johnson, P., Buehring, A., Cassell, C. and Symon, G. (2006) ‘Evaluating qualitative management research: Towards a contingent criteriology’, International Journal of Management Reviews, 8(3), pp. 131–156.
I agree with Peter Jackson and others that calculating reliability inherently requires two independent measures, so there is not way that this could be done by a single individual.
But I also wonder why you feel that you need to calculate some form of reliability. In some fields, such as communications studies, this is typically expected, but in most fields it is not. In particular, procedures such Thematic Analysis are inherently subjective, and are widely accepted within qualitative research on that basis.
Due to the subjective nature of qualitative research, it is incumbent upon the researcher to apply several strategies to ensure that the information obtained truly captures the voice of the population of interest (Lietz, Langer & Furman, 2010). The term “rigor” in qualitative research is analogous to achieving ‘validity’ in quantitative circles. According to Lietz, et.al. (2010), rigor suggests that the research has been subjected to efforts that increase confidence that findings are representative of the meanings presented by our participants. These efforts should place awareness on how we interpret data in order to minimize reactivity, bias and in essence place priority on the voices of the participants. The term 'trustworthiness' is utilized in qual research and that is the goal to achieve, so, if you are not part of a team and are the only rater - then you need to have a variety of rigorous initiatives in your design directed towards establishing trustworthiness. As already mentioned, member checking (Charmaz, 2011) is one way of doing this as well as establishing a paper trail (Charmaz, 1990), being cognizant of your 'positionality' and being reflexive in the process. A good qualitative research text such as by Patton (2002) will illustrate these methods of establishing trustworthiness in great detail.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Coding