I agree that the most likely way to accomplish your goal i to reduce the comparisons into a 2x2 table based on whether each of two coders reports the code as present or absent for each passage. 

I also think that Kappa would be just fine, especially if the reviewers requested it. Krippendorff's alpha is as much a conceptual framework as an actual measure, but either coefficient is easily calculated in the simplest case of a 2 x 2 table of agreement and disagreement.

Many thanks for your helpful responses. If I was to reduce the comparisons into 2x2 tables (coder X code), would this mean running a separate test for each code individually? Is this conceptually sound?

You can produce a set of 2x2 crosstab (coder X code) - after you have dichotomized each variable into a binary variable - and execute the Cohen's Kappa for each code individually. Later you can also think to compute group of answers for specific themes

You can report the average Kappa and the range for the main set of codes that you use in your analysis.

The use of Kappa requires minimal assumptions about the underlying nature of the data. Based on Haley & Osberg, 1989, your data collection conditions for the use of Kappa must be:

1. The subjects to be rated are independent of each other

2. The raters score the subjects in an independent fashion

3. The rating categories are mutually exclusive and exhaustive

Kappa can be used with multiple ratings per subject test in which ratings can be performed by the same set of raters or by different raters. Measurements of agreement between raters are from the agreement of observed values when the responses are categorical. With a normal distribution, the Z statistic can be used to test the null hypothesis that kappa is equal to 0.

You could also map Kappa value to the Landis and Koch-Kappa’s Benchmark scale table to support your finding. Landis and Koch, 1977, proposed a benchmarking scale known as “Landis and Koch-Kappa’s Benchmark Scale”.

For most purposes, values greater than 0.75 may be taken to represent substantial agreement beyond chance, and values between 0.40 and 0.75 may be taken to represent moderate agreement beyond chance based on Landis & Koch Benchmark. You could cover or have two approaches. The first approach could present that raters are in agreements. The second approach is to present the similarities of reviewers’ answers/input between raters. Reviewers’ answers then coded into numeric values (you could use Excel table for this data entry). Converting the ordinal scale to its numerical equivalent, and analyze the data as if they were simple numerical data. The codes for ratings were: Yes/Agree = 2, Possibly/Partially = 1, No/ Disagree= 0.

Also, to assess Inter-rater reliability (IRR), use the Intra-class Correlations Coefficients (ICC). The ICC is one of the most commonly used statistics for assessing IRR for either interval or ratio scale of measurement, and it is suitable for studies with two or more raters, and may be used when all subjects in a study are rated by multiple raters. In general practice, research confidence levels testing (CI) could be used for lower and upper bound limit. The 95% CI of .85 to .90 means that the researcher can be 95% confidence that with repeated action on a sample, the true sample statistic is in the range of .85 to .90. Statistically, a lower ICC value is associated with large magnitude of disagreements, and conversely, a higher ICC value shows a greater IRR.

If the number of raters is fixed, and the subjects are chosen randomly, then you use two-way mixed in the calculation.

Cochran’s formula can be used to determine sample size. The true margin of risk is based on the central limit theorem, which states that the distribution of random samples from any population, when sufficiently large (N>=30), tends to be close to the normal distribution. Based on Arjomand 2002, the larger the sample size, the more pronounced this phenomenon becomes.

As others in this thread recommended the coding and other references, I wanted to expand on their recommendations and give you a little more detail information and mention something about sample size.

In short, it is all about coding the reviewers’ input in numerical values (possibly in an Excel table), and then using a Statistical Package (such as SPSS, or R) to get Kappa value.

Gwet, K. L. (2012). Handbook of inter-rater reliability (3rd ed.). Gaithersburg, MD. Advanced Analytics LLC.

Haley, S. M., & Osberg, J. S. (1989). Kappa coefficient calculation using multiple ratings per subject: a special communication. Physical Therapy, 69(11), 970-974.

Landis, J. R., & Koch, G. G. (1977). An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics, 363-374.

An inter-observer concordance index of categorical or quantitative multivariate data is the iota coefficient (ι). Gamer, Lemon, and Fellow have implemented the code of Janson and Olsson in the iota function (ι) in the package irr in R which calculates just the iota index (ι).

One of the most interesting properties of the iota coefficient, in contrast to other common-use agreement coefficients, lies in the possibility of being used to assess the agreement in the multivariate case, since it allows operating with two -or more- characteristics or behaviours at the same time in the form of two -or more- sets of items, which supposedly represent a multivariate construct.

- Harald Janson and Ulf Olsson. (2001). "A Measure of Agreement for Interval or Nominal Multivariate Observations". Educational and Psychological Measurement, Vol. 61 No. 2, April 2001 277- 289.

- Matthias Gamer, Jim Lemon, and Ian Fellows Puspendra Singh. 2010. irr: Various Coefficients of Interrater Reliability and Agreement. R package version 0.83. http://CRAN.R-project.org/ package=irr

Translated with www.DeepL.com/Translator