In a study, we conducted semi-structured interviews with people living with HIV and then analysed their transcription logs thematically by using inductive content analysis methodology. The approach is recommended if former knowledge is inadequate or fragmented about a phenomenon. Accordingly, the explanatory or thematic categories are derived from the data. That is to say that we did not have pre-determined codes; we derived the thematic framework from the data.
We sent our study report to a journal recently. The reviewers have required us to report how we ensured the inter-rater reliability. In the analysis phase, two researchers performed the open coding individually and then compared their findings. After iterative discussions, they reconciled their thematic sets consensually. However, they did not calculate the level of agreement between each other statistically.
I would be glad if you could share your opinions and knowledge on calculating inter-rater reliability (Cohen's Kappa) in qualitative inquiry such as thematic, content and discourse analyses or grounded theory.
Thanks in advance.
Hello Mustafa Volkan Kavas
You may find this work useful to you.
https://doi.org/10.1002/9781118445112.stat00365.pub2
Good luck
In general, inter-rater reliability is most appropriate for the use of deductive rather than inductive coding systems. Such deductive approaches rely on the application of pre-determined coding systems that are typically based on manifest, rather than latent content.
In other words, inter-rater reliability would not be appropriate for the kind open coding that you used, and this would be recognized by the reviewers of a journal that routinely published qualitative research. Unfortunately, however, it seldom makes any difference to argue with a reviewer, no matter how poorly informed they are.
If you want to continuing working with this journal, then you can find any number of free calculators for Cohen's Kappa through a simple Google search. To use these calculators, the first thing you need to do is to "unitize" you data, so that both coders are applying their codes to the same segments of text. Next, you need to produce a list of codes that each coder applies independently to the units.
From there, you need to to cross-tabulate the codes that coder #1 made on the rows and Coder #2 made on the columns. This will generate a square table where the diagonal captures the number of times both coders agreed to apply that particular code to a given text segment. Once you have that table, you can use it to get a kappa coefficient by inputting it to a calculator, such as:
https://www.graphpad.com/quickcalcs/kappa1/
Step 1: Calculate po (the observed proportional agreement): 20 images were rated Yes by both. 15 images were rated No by both. So, Po = number in agreement / total = (20 + 15) / 50 = 0.70.
Step 2: Find the probability that the raters would randomly both say Yes. Rater A said Yes to 25/50 images, or 50%(0.5). Rater B said Yes to 30/50 images, or 60%(0.6). The total probability of the raters both saying Yes randomly is: 0.5 * 0.6 = 0.30.
Step 3: Calculate the probability that the raters would randomly both say No. Rater A said No to 25/50 images, or 50%(0.5). Rater B said No to 20/50 images, or 40%(0.6). The total probability of the raters both saying No randomly is: 0.5 * 0.4 = 0.20.
Step 4: Calculate Pe. Add your answers from Step 2 and 3 to get the overall probability that the raters would randomly agree. Pe = 0.30 + 0.20 = 0.50.
Step 5: Insert your calculations into the formula and solve:
k = (Po – pe) / (1 – pe = (0.70 – 0.50) / (1 – 0.50) = 0.40.
k = 0.40, which indicates fair agreement.
Source: https://www.statisticshowto.com/cohens-kappa-statistic/
Good luck!
The answer from M.K. S Al-Mhdawi only works if you have a single code, which results in a 2 x 2 table (i.e., yes or no from each coder). If you more than one code, you need to create an N x N square table for all your codes (plus one additional row and column for "no code").
One can use NVivo to calculate the Cohen's kappa coefficient in qualitative analysis.
Hello, if you use nvivo here is a tutorial
http://help-nv11.qsrinternational.com/desktop/procedures/run_a_coding_comparison_query.htm
Have a nice day
MAXQDA and Dedoose will also calculate Kappa, while ATLAS.ti calculates an alternative known as Krippendorff's alpha. In each case, you need to apply the same set of codes to unitized transcripts, so that the separate codings can be compared.
Thank you very much for your responses. We have benefited from them a lot.
- Badges
- Science topic