Hello Rumeysa Ozturk

The Inter-Rater Reliability (IRR) refers to the degree of agreement between raters. Generally, researchers used Cohen’s Kappa to calculate inter-rater reliability. Cohen’s Kappa formula calculates the percentage of items that the raters agree on, while accounting for the fact that the raters may happen to agree on some items purely by chance.

The formula for Cohen’s kappa is calculated as:

k = (po – pe) / (1 – pe)

where:

k=kappa’s coefficient

po: Relative observed agreement among raters

pe: Hypothetical probability of chance agreement

Cohen’s Kappa (k) always ranges between 0 and 1, with 0 indicating no agreement between raters and 1 indicating perfect agreement between raters. An inter-rater reliability of 60% may be acceptable. However, some accept K> 51 and some required high level of inter-rater reliability (kappa) in medical settings.

This a good explanation of Cohen’s kappa formula:

https://www.real-statistics.com/reliability/interrater-reliability/cohens-kappa/

You may also check this article:

McHugh ML. Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22:276-282:

https://www.biochemia-medica.com/en/journal/22/3/10.11613/BM.2012.031

I personally used Cohen’s kappa!

• Another option is to use the weighted Cohen’s kappa, which takes into account the degree of disagreement between the raters.

This a good explanation of weighted Cohen’s kappa:

https://www.real-statistics.com/reliability/interrater-reliability/weighted-cohens-kappa/

Hi Salah Ahmed, thank you very much for sharing this information. I appreciate it! I am specifically trying to understand how to accommodate multiple coding pairs. Do you know if it would make more sense to look at the interrater reliability separately for each coding pair or is there a way to evaluate them together? Thank you very much.