Have a look into intraclass correlation coefficient...

You may search using Krippendorf's alpha, which works for any kind of rating (qualitative, quantitative...) and, IIRC, any number of raters.

I think Fleiss kappa is better than Cohen's kappa, since it is really based on agreement probabilities, wheras Cohen's kappa is based on the chi-square of independance value.

But, if raters are different for every subject, it is I think non standard and you may have to develop specific details... Or to precise your question. If some raters rate only one individual, it may be difficult to conclude anything...

I understand that you have N subjects, and K raters, where the n_r number of subjects rated rate by rater r is generally small ("a number of"), and the k_s number of raters of subject s varies between 2 to 6. I understand you want to compare the raters, and characterising the subjects is not an objective. Is this correct? Are the three categories ordered or nominal?

It looks like a multi-rater multi-reader problem for categorical data. As you mentioned Gwet AC1 is an alternative to Kappa. He claims some advantages of AC1 over kappa, as you noticed, and shows some empirical evidence with simulations studies. IF you didnt read so far, I recommend you take a look at Gwet's book. Either at his website (http://agreestat.com/) or at google books. He comments over several inter-rater measures including Kappa AC1 and others. There are simulations results which leads to believe that AC1 is less prompt to bias and less prompt to the kappa paradox. Showing both wouldn't be that bad as many are not familiar with AC1. At his website there several SAS macros to estimate several reliability measures including multi-rater multi-reader scenario.

I think thoughtful people can get on board with an overall kappa (across pairs of raters) of some variety if appropriately cast in terms of the implications both computationally and with regards to the scope of your assertions. Here's where clear writing, thoughtfullness, and sales should meet to avoid "throwing the baby out with the bath water." ...assuming, after your carefull consideration you haven't discovered it's all bath water. Were I reviewing the manuscript, I would pay special attention to who the pool of raters were, what the combinations were that were most frequent, why subjects might have more pairs, and what kind of biases these may implicate in the data. For example, is this naturalistic date where the cases with more than 2 pairings were a result of some challenge in the ratings and more opinions were sought? What was Tukey's quote? paraphrasing: "Better an approximate answer to the right question than an exact one to the wrong question" ...I'd supplement with "or no answer at all". Just be fair and convincing.

Probably Krippendorf's alpha is what is required here...it creates a coincidence matrix to control for agreement by chance. Implemented as a download in either spss or R

A few references to consider:

Endicott, Tracy, Burt, Olson, and Coccaro (2002). A novel approach to assess inter-rater reliability in the use of the Overt Aggression Scale-Modified. Psychiatry Research, 112, 153-159.

Krippendorff's alpha: http://www.asc.upenn.edu/usr/krippendorff/dogs.html

http://www.asc.upenn.edu/usr/krippendorff/dogs.html

It seems to me that the key question is what conclusion you are trying to test. If the comclusion does not depend on which measure you use, it would be nice to be able to say that. If the conclusion DOES depend on the measure, you really are obliged to say so, and to try to figure out why. That said, my guess is that the choice of measure does not matter, and I would probably just go with Kappa, because it's widely used and well understood.

I basically agree with Ken. The question to me is not so much a statistical question as "what are you trying to find out?" If the question is how popular or useful, or something like that, are the different subjects compared to each other, or if it is 'how much variance is there due to different raters", the problem of what statistic to use differs. In the latter case, knowing what it is you want to be able to predict, or reproduce, or if you are really comparing subjects rather than raters, the answers are different. If there is one rating scale, or perhaps several, then the importance of the statistic used may or may not be important depending on what the question is, and how generalizeable you want it to be. Of course I am not a "real" statistician, although I was trained by one and mentored by another, but I am a real researcher. As such I want to know what I am trying to find out, what use will the results be put to, what is the cost if the statistically based judgement is "yes" versus "no", what is the matching benefit, and so forth. I don't believe "statistics" can ever be used with true validity and integrity with just the number of levels or categories there are in dependent and independent variables. Without more detail about what the questions are and to what purpose the answer will be put, a fairly well-known, and relatively assumption-free Kappa works as well as anything. If the results are basically "pilot" efforts, intended to point in certain "in the wilds" ways to direct the next little piece of a research program, versus "proof of", then obviously the choice of statistic is less an issue. Significance just tells you how much confidence to have in the results, and if the questions are not good, clear ones, the most sensitive statistic and strongest results will be meaningless or misleading.

Thanks for the many helpful and stimulating comments so quickly received.

(Gabor) The 3 categories are ordinal.

(Ken, Barbara) The aim of the investigation is to describe and assess how well the various raters agree with each other, and thus how reliable the final joint decision, based on the independent ratings, is. A 'gold standard' is not available.

http://john-uebersax.com/stat/agree.htm. is wonderful site having all types of inter rater agreement analysis. I agree with Johannes Borlin

I agree with much that has been offered, though I think you need to push a bit further to consider that "How well different raters agree" depends fundamentally on whether they are looking and judging the same cases. Where some are looking and judging some cases but not others, their agreement with other raters is contingent on what they rated in common, right? Nonetheless, if you have enough common cases among the raters to analyze, that is, enough to generalize that the tendency for any rater is to be one way or another, which is a question of statistical power, otherwise you have a weak basis for assessing some raters, and maybe a strong one for others, and overall, a weak or strong basis for judging the extent of agreement, depending on the power provided in the number of common cases. You may need to drop some raters due to too few common cases, and assess only those with a minmum number. Given your unknown power issues, I have found Fleiss's Kappa quite satisfactory for assessing agreement beyond chance expectations between raters of similar cases.. I have published on clinical abstraction reliability and you can find a copy of that work on my page if interested. Best of luck.

Armitage and Berry statistical textbook describe kappa very well, but I just can't find it at present so maybe I've loaned it out more then likely and it hasn't been returned. Sorry.

Krippendorf's alpha takes care of multiple raters, as well as missing labels. See: https://en.wikipedia.org/wiki/Krippendorff's_alpha and the irr package in R.

I have a somewhat similar question to the original poster.  In my data, there are roughly 400 different patients, each of whom was rated on a continuous scaled test by two different raters.  The trouble is, there are many different raters.  So, I have two different scores per patient, many the same, some slightly different-how could I characterize the rate of agreement in the scores overall?  Also, any thoughts on setting up the data frame for analysis would be helpful.

Jerry

In case it may be of interest, here are links to open-source articles that discuss a non-parametric machine-learning algorithm used to assess inter-rater or inter-device agreement.

• https://odajournal.com/2014/04/13/how-to-assess-inter-observer-reliability-of-ratings-made-on-ordinal-scales-evaluating-and-comparing-the-emergency-severity-index-version-3-and-canadian-triage-acuity-scale
• https://odajournal.com/2014/04/14/how-to-assess-the-inter-method-parallel-forms-reliability-of-ratings-made-on-ordinal-scales-emergency-severity-index-version-3-and-canadian-triage-acuity-scale
• https://odajournal.com/2015/04/21/estimating-inter-rater-reliability-using-pooled-data-induces-paradoxical-confounding-an-example-involving-emergency-severity-index-triage-ratings
• https://odajournal.com/2016/11/11/novometric-vs-oda-reliability-analysis-vs-polychoric-correlation-with-relaxed-distributional-assumptions-inter-rater-reliability-of-independent-ratings-of-plant-health

Fleiss Kappa.

SPSS extension to do it can be found here (I tried it and it works seamlessly):

https://www.ibm.com/developerworks/community/files/app#/file/48234a16-fb14-4bee-8c18-570319c57108

It is possible that my method of data analysis, Statistical Implicative Analysis, applied to ordinal variables answers your question (Springer, 2008)

Good pm wha can be the best inter rater reliability test can be done if I only have four raters. Thank you.

You can consider Gwet's AC1 using R program. Gwet's AC1 is an alternative to Fleiss' kappa which critics of prevalence bias (low kappa value despite high percent agreement) while Gwet's AC1 is more stable in estimating inter-rater reliability of a scale. The R function for Gwet's AC1 statistics is as below:

Gwet, K. L. (2010). R functions for calculating agreement coefficients. [Online] Available from:http://agreestat.com/r_functions.html [Accessed 25th March 2018].

This R function also gives Fleiss' kappa values.