Doesn't sound like a bad situation, really. Two questions, though:

1. When you say "coded," do you mean that they assigned some sort of rating or "score" (such as how positive vs. negative the memory was on, say, a 10-point scale)? Or that they assigned a number of separate qualitative codes (such as presence/absence of excitement, anger, fear, love, etc.)?

2. Do you intend to work with each memory separately or combine them into a single variable?

Let's look at each situation in turn.

If the ratings resemble scores, then you can either calculate the intraclass correlation coefficient (ICC), which is available in SPSS as an option under "scale - reliability," or correlate the scores assigned by the 3 raters (A-B, A-C, and B-C), take the mean, and use the Spearman-Brown prophecy formula to adjust the result. (Easily done by hand: (1) multiply the mean correlation times the number of judges - in this case, three; (2) subtract from 1 the product of the mean correlation times one less than the number of judges - in this case, 2; (3) divide the result from step #1 by the result from step #2. The two procedures generally yield similar, though not identical, results.

If the results are along lines of either/or codes (categories), then you'll want to use Cohen's kappa. There are online calculators for this that might be easier than SPSS, but it's up to you.

If each memory is to be treated as a separate variable, then you have 3 inter-rater reliability calculations to perform: one for each variable. That's it!

If the memory codes/ratings are being combined into one comprehensive variable, then you have just one inter-rater problem, but you need to do the "merger" first. You might also want to look at how strongly the 3 memories correlate with one another within each rater's work (internal consistency).

Doesn't sound like a bad situation, really. Two questions, though:

1. When you say "coded," do you mean that they assigned some sort of rating or "score" (such as how positive vs. negative the memory was on, say, a 10-point scale)? Or that they assigned a number of separate qualitative codes (such as presence/absence of excitement, anger, fear, love, etc.)?

2. Do you intend to work with each memory separately or combine them into a single variable?

Let's look at each situation in turn.

If the ratings resemble scores, then you can either calculate the intraclass correlation coefficient (ICC), which is available in SPSS as an option under "scale - reliability," or correlate the scores assigned by the 3 raters (A-B, A-C, and B-C), take the mean, and use the Spearman-Brown prophecy formula to adjust the result. (Easily done by hand: (1) multiply the mean correlation times the number of judges - in this case, three; (2) subtract from 1 the product of the mean correlation times one less than the number of judges - in this case, 2; (3) divide the result from step #1 by the result from step #2. The two procedures generally yield similar, though not identical, results.

If the results are along lines of either/or codes (categories), then you'll want to use Cohen's kappa. There are online calculators for this that might be easier than SPSS, but it's up to you.

If each memory is to be treated as a separate variable, then you have 3 inter-rater reliability calculations to perform: one for each variable. That's it!

If the memory codes/ratings are being combined into one comprehensive variable, then you have just one inter-rater problem, but you need to do the "merger" first. You might also want to look at how strongly the 3 memories correlate with one another within each rater's work (internal consistency).

Tuğçe Koç I am not sure if you define your question properly or I do not understand the question.

What I understand is that your three columns represent three variables. If this is true, you should measure the correlations between the variables which are based on rating first and second-rater. High correlations should signify high reliability.

Hello, Stephen Joy

Does the content in each column represent what was coded? If so, then it should not matter how many columns you have. You still would be testing reliability based on each item for which your coders coded. In any case, for calculating reliability, this is a good tutorial article that you might find helpful:

Process guidelines for establishing intercoder reliability in qualitative studies

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.857.2280&rep=rep1&type=pdf

Wishing you the best.

Brooke Hallowell

Hello Brooke Hallowell -

I assumed that each column represented the codes/ratings made by one judge, so the question was how to calculate inter-rater reliability across three judges. What I was unsure of was whether the data reflected categorical codes (as in a qualitative rating system) or quantitative ratings of some sort. Since Tuğçe Koç has not replied to any of our responses, that remains undetermined. But if her work is essentially qualitative, then the tutorial you provide may help her out!

Oh, yes, Stephen Joy

I actually meant to address my comment to her and mention you but some text inadvertently got deleted. In any case, you and I are on the same page, and I hope that Tuğçe Koç is feeling clarity!

Thank you for your answers dear Stephen Joy Brooke Hallowell Senad Bećirović .

I will try to explain more into detail by using a picture from my SPSS data.

We asked each participant to write 3 memories for the word "SUCCESS". Therefore, we have 3 memories for each participant. Then, 2 raters coded these memories on a Likert-scale (between 1-3) according to spesificity (1=memory is not specific, 2=memory is moderately specific, 3=memory is specific). Now, we have 3 columns for Rater 1 and 3 columns for Rater 2.

In this case, we want to calculate interrater reliability for each memory. Should we calculate interrater reliability separately (I mean should we have 3 different alphas for each memories) or should we calculate mean first and then calculate interrater reliability (I mean take a mean for 3 memories and then have 2 different means for 2 different raters)? In the first scenario, we will have 3 different alphas. In the second scenario, we will have only one alpha.

Which way you suggest? And which type of calculation should we use? (Cohen's Alpha, Krippendorff’s Alpha or Intra-class correlation coefficient?

thank you for your help

Best Regards

I think you have 2-3 options here, any one of which can be justified.

1. Within each rater, create an overall specificity for each participant score by summing the scores from the 3 memories. Then run a single intraclass correlation coefficient (ICC) across the 3 raters. This has the virtue of simplicity, and if you will be combining the scores across memories in other analyses, it's the way to go. Scores, then, could range from 1-9 within each rater.

2. Run separate ICCs for each memory. The problem here is that since it's just a 3-point scale there may not be all that much variability. But if you are treating each memory as a distinct data point in other analyses, it would make sense. Though due to the very few response options, #3 might make more sense.

3. Run Krippendorf's alpha for each of the 3 memories. The thing about doing this is that you are demanding perfect agreement across raters; if they don't converge exactly, it will count as a case of disagreement. So I would NOT do this if you are combining scores across memories - the odds against perfect agreement, even if the judges are doing a good job, are very long.

I personally don't care whether you use the ICC or the Spearman-Brown approach in case #1, or whether you use a Cohen's kappa variant or Krippendorf in case #2. If someone else has strong opinions there (especially someone on your review committee), go with them!