I am doing research on the concept of subalternity in rural women of Sindh. I collected data from literature (fictional stories) and real life (life story interviews). I applied the method of deconstruction to analyse the data. To be honest to my study and to achieve replicability I presented the data analysis process. I mean I presented the process of coding with textual references and major and monior themes, textual clashes, etc.
During the analysis of this data, I took help from a researcher to assess my coding process. It means I used inter-coder.
Now, my external is suggesting me to discuss the inter-coding reliability. Kindly assist me that how should I discuss it with issues of agreement , disagreement, etc.
Awaiting for your response.
To say a bit more about the process that Peter Kindle describes, you need to begin with code-able units so that you each are applying your codes to the same thing (e.g., text segments). When this is done, it implies a 2x2 table of agreements and disagreements for each code, from which you get the overall agreement rate.
Taking this a step further, you can calculate one of the coefficients of inter-rater reliability to take into account the "chance rate" of agreement and disagreement. The most widely used of these is Cohen's Kappa -- although some people argue that there are better alternatives, such as Krippendorff's alpha.
Hi, Sabah,
Interrater reliability in qualitative studies is usually reported as simple percentages of agreement. for example, if your collaborator reviewed 30% of your data set and his/her coding agreed with yours 65% of the time before reconciliation and 98% after reconciliation, then this is generally all that is required. If the second coder changed the coding in any way, of course, you would also need to report this.
To say a bit more about the process that Peter Kindle describes, you need to begin with code-able units so that you each are applying your codes to the same thing (e.g., text segments). When this is done, it implies a 2x2 table of agreements and disagreements for each code, from which you get the overall agreement rate.
Taking this a step further, you can calculate one of the coefficients of inter-rater reliability to take into account the "chance rate" of agreement and disagreement. The most widely used of these is Cohen's Kappa -- although some people argue that there are better alternatives, such as Krippendorff's alpha.
Dear Sir Kindle and David,
I cannot use any scale or index (alpha, coefficient, etc.) because my study is ethnographic in nature where I am dealing with sensitive cases including rape, sexual assault, honor killing and forced marriage of women.
I do not know how you can do anything with regard to inter-coder reliability unless you do some kind of calculation on the rate of agreement between two coders
If the other researcher who worked with you actually did code your data (rather than just "review" your coding), then you have everything you need. If not, then you will need to discuss this issue with your external reviewer.
Dear Sabah,
Do you mean he (the other researcher you consulted) assessed your coding process or you both coded the data independently? If you have both not coded the data independently, then I don't think you can talk about inter-rater or inter-coder reliability here. I say this because, inter-rater reliability is mainly a test to show the level of agreement between two or more coders [in this case you and the researcher you sought help from] after they have both independently coded the data. David Morgan earlier mentioned Cohen's alpha and Krippendorff's alpha, both of which are widely used.
However, what you can do (that is, if the coding was not independently done by both of you) is that, you can talk about you making attempts to ensure reliability of your codes/findings by describing the coding process into details to the other researcher (of course, also describing it in the methods) for him to assess the process and the codes. I don't know if you have done re-coding after his assessments. If so, you describe how it was done and how you have resolved the inconsistencies or disagreements relating to the codes, if any. I hope this will help you.
- Badges
- Science method
- Similar topics
- Methodology
- Research Methodology
- Qualitative Research