Hi Hubert,

What is your outcome? If your outcome is categorical, you may consider multi-rater kappa. Here is a reference: http://www.ajronline.org/doi/full/10.2214/ajr.184.5.01841391

If it's continuous measures, try ICC (intra-class correlation coefficient)

Thanks Fang-Yong for your comment. Unfortunately, I have a constraint to respect in using Bland Altman as most of the studies in the fied use such approach. I need to allow comparisons and so to conform BA analysis.

Actually, three readers measure items; therefore I am considering continous values. In addition, my BA display reports vertically the relative difference instead of the absolute difference.

Hi Hubert,

Can you give us a bit more info on what you're trying to do? I'm not familiar with the expression "three readers circularly involved". What's your data type, number of measurements/raters and repeats?

In general for stats of agreement:

Cohen's kappa gives the agreement coefficient for nominal grades and each outcome is binary (agreement or not agreement) [1-3]. If you want to take into account the degree of agreement or disagreement you can use a weighted kappa (with linear or quadratic weights usually). Both kappas are corrected for chance [1].

For continuous variables you need either the intra-class correlation coefficient ICC [8] or Bland-Altman BA [7].

Note that for ordinal data the ICC and weighted kappa with quadratic weighting are comparable (and exactly equivalent for uniform marginal distributions [8,10]).

BA plots is a graphical representation of the same concept, showing for two assessors (or groups of assessors) the difference for each assessment against the mean of each assessment [4, 11]. As such it shows any bias and the limits of agreement between two raters.

1 Cohen J. Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213-20.

2 Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Meas. 1960;20:207-16.

3 Kraemer HC. Extension of the Kappa-Coefficient. Biometrics. 1980;36:207-16.

4 Bland JM, Altman DG. Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement. Lancet. 1986;1:307-10.

5 Landis JR, Koch GG. An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics. 1977;33:363-74.

6 Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159-74.

7 Rousson V, Gasser T, Seifert B. Assessing intrarater, interrater and test-retest reliability of continuous measurements. Stat Med. 2002;21:3431-46.

8 Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86:420-8.

9 Fleiss JL, Cohen J. Equivalence of Weighted Kappa and Intraclass Correlation Coefficient as Measures of Reliability. Educ Psychol Meas. 1973;33:613-9.

10 Fleiss JL, Shrout PE. Approximate Interval Estimation for a Certain Intraclass Correlation-Coefficient. Psychometrika. 1978;43:259-62.

11 Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Int J Nurs Stud. 2010;47:931-6.

Hi Virginie,

Thank you very much for all these documentations. Here are more details about my concerns.

I would like to setup experiments investigating repeatability of measurement (continuous evaluations as in millimeters) in a similar way as: http://www.ncbi.nlm.nih.gov/pubmed/23489982

So 3 readers will repeat measurements on a sample size of about 100 samples. BA analysis will be carried out considering readers 2 by 2 (Reader1 against Reader2, Reader1 against Reader3 and Reader2 against Reader3), for each couple of readers, Limit of Agreement can be computed with CI. Finally, I would like to average the 3 Limits of agreement previously computed but I don’t know how to "average" the CI.

I hope I succeed in clarifying.

Hi Hubert,

Thank you that is very clear. If you have no reason to trust one rater more than another (no one is assume "gold standard" etc), then you can present an average ICC for continuous data, where you average the values of ICC you get for each possible pair of raters. Similarly, you can present an average of your limits of agreement from a BA analysis as you suggest.

Now for the CI associated to that average limit of agreement from BA, you should combine the errors by following the propagation of error rules, a good summary of which can be found here: http://ipl.physics.harvard.edu/wp-uploads/2013/03/PS3_Error_Propagation_sp13.pdf

In practice for this case you add in quadrature (since you need the formula for the sum in calculating the average) then divide the result by the number of possible rater pairs. Example: the combined error for Q where Q is the average of measurements a, b and c such as Q= (a+b+c)/3, with errors associated to a, b and c being respectively da, db and dc (so the confidence interval of a would be [a-da, a+da]):

dQ = 1/3 [(da*da + db*db)^1/2]

I hope this helps !

Thank you very much Virginie, it seems that's exactly what I was looking for. This was great help