Paulo -

Do you mean that RR1and RR2 readings were taken at the same time on a given patient? You could plot the RR1 and RR2 values on a scatterplot, if they were taken at the same time for the same patient, so RR1i and RR2i match for each i, and you can plot them as (x,y) points. If RR2 is y, then a regression through those points would be "predicting" RR2 from RR1, the x-values. It sounds like it might be a linear regression through the origin, where the slope b would be approximately 1, if the two RR measures are approximately equivalent. The standard error of the slope is important. It will help indicate how reliably good this comparison may be. This would be heteroscedastic, as the "error" (estimated residual) in "predicting" (estimating) bigger y's from bigger x's would also be larger. A classical ratio estimator will likely do well, where slope is just the sum of the y's divided by the sum of the x's. (I think that the appropriate information could be extracted for making such a regression line at the attached link.)

Cheers - Jim 

Data CRE Prediction 'Bounds' and Graphs Example for Section 4 of ...

Hello, Paulo.

You should use ICC (2,1), AKA ICC-agreement. The nomenclature employed by SPSS: Two way, random effects; absolute agreement; single measures.

Lin´s coefficient is likely to show the same value than ICC.

In my opinion, the best choice is the use of ICC and the Bland-Altman method: While ICC is a global measurement of reliability, Bland-Altman method provides information about the distribution of differences of measurements.

Bear in mind that ICC requires normality of distribution. Moreover, it shows a paradoxical inflation when used on heterogeneous samples, with a wide range of values. That´s why it should not be employed as the sole statistic.

Hope it helps.

Thank you both!

James R Knaub, for a single patient I can have a long reading (24 hours). RR1 is from an instrument (I1) and RR2 is from another instrument (I2). I actually want to measure the same (RR) from 2 instruments, at the same time, and want to check if the information coming from I1 is statistically equivalent to I2, so I will be able to use I1 or I2...

Emilio Pariente-Rodrigo, I must use Matlab (I am a Mac OS X user) to program the ICC (2,1). So, if I understood correctly, in your opinion, for the reliability assessment I must use ICC or CCC and to reach the agreement I must use Bland-Altman method. Is this correct?

Dear Paulo:

ICC, CCC and Bland-Altman are methods to assess reliability with continuous data.

CCC is identical to some ICC.

ICC has several advantages: global assessment of reliability since it combines both bias error and random variation into a single coefficient; besides that, it is possible to evaluate the agreement when there are more than 2 raters.

The main disadvantage: ICC is quite affected by the heterogeneity of the sample and commonly shows very high values that may be biased (In a recent study, we found ICC values of 0.58 -first tertile-, 0.28 -second tertile- and 0.46 -third tertile-, whereas ICC on the complete series was 0.90!!)

The Bland-Altman plot is a good method to know the distribution of differences, the heterokedasticity, limits of agreement and repeteability coefficient.

My suggestion is the use of both the ICC and the Bland-Altman method.

Regards.

Sounds like regression may be a good tool here.  You would be looking at values for b, standard error of b, and perhaps the variance of the prediction error for WLS.  You could look at that as well as what Emilio has provided.  -  A follow up posting by you of your results should be interesting. -  Best wishes. 

Thank you both, again!

I am starting this subject, so this is not easy for me...

I also must separate with tools are appropriate to reliability and for agreement! Not easy. 

Emilio, when you wrote «it is possible to evaluate the agreement when there are more than 2 raters.», you mean I also could evaluate the agreement if I have more than 2 instruments?

I must understand how ICC(2,1) works since I have to program it in matlab!

James, do you mean multiple regression? Or your suggestion is like your 1st answer?

Yes, you may calculate ICC with more than 2 measurements (raters or instruments)

To fully understand the 6 variants of ICC, have a look at this file.

Good luck.

Paulo - 

It would just be one regressor, as you would be picking which one of two instruments to be acting as the predictor (x) for the other one (y).   So with only one regressor (x), this would not be multiple regression - so the first answer. 

Note that it really isn't a question of "Do these instruments perform identically?"  It is more "How do they differ; to what degree; and where?"  Scatterplots can be very informative in your quest to make your comparison.  Graphics can be very helpful. 

Jim 

Paulo -

Please note that when the variance of the prediction error is used to make confidence bounds about the predicted y-values on a scatterplot, the values for y are to be read vertically on the scatterplot.

That is, for a given reading from instrument x, you read vertically to the upper and lower confidence bound curves to see the confidence interval for the corresponding predicted y-value. So for a given reading on instrument x, you will see a confidence interval on which a corresponding reading from instrument y might fall.  If the difference between those x and y numbers, in a given x,y pair, for a given range of x, is not satisfactory for practical purposes, then there will be a problem in substituting one instrument for the other.  

Even if you do not look at confidence bounds (though I recommend that you do), it would still be useful to plot your points ((x,y) observations) on a scatterplot. You could color code points from different patients. 

I think we are clear on this, but I just want to emphasize that the (x,y) points are for the two instruments used at the same time, for a give moment's reading on a given patient. 

Best Wishes - Jim 

Hello all, 

I am back with some questions about the same issue.

According to Intraclass Correlations: Uses in Assessing Rater Reliability (Shrout, 1979), image attached, I have a question about the kind of study it is and with type ICC I must use. 

According to the subject exposed, I want to compare the same (heart) beat thought 2 methods. So in this case what is the target, the raters and the judges?

target =?

raters = beat ?

judges = the methods ?

I appreciate if someone could help me. 

Paulo

Paulo -

I am certain that Emilio can answer much better than I can, but from the information you sent, which was not familiar to me, it looks like the "targets" would be the patients here. Also, you said, in the question above that you have two different instruments running simultaneously on the same patients (n of them) making corresponding readings on those n patients, correct? And these two machines, I now take it, are of the same type, just two different judges (k=2), correct?

I think the raters are the judges, which are the two machines.

If by some miracle I got this right, then it sounds like case 2, which would be better anyway, and I think that is what Emilio has above. It seems to be based on more complete information by including the rj error term.

Jim

You know, k=2 is very small, and if you only have the two instruments and aren't looking to see how others like them will perform in the future, but just want to know if you can use the two you have interchangeably , then you might want to do something very simple. If you have data collected on two instruments simultaneously (ie, paired data), you could plot them together. Graphics can be very useful. You can see quickly how often they may vary and by how much. You can see if one tends to stay lower than the other, and perhaps occasionally is much larger, or whatever the case may be. - And you can take the difference in measurements for each pair and do some descriptive statistics on those differences. - It all depends on what you really want to know.

So according to my issue, when I could have the case 3 (all possible raters rate each target)?

Because each of 2 methods will rate each patient ...

It was not clear at all to me from that newer paper, but from what i looked at in the the older one I got the impression that case three was to be used if you had the entire population of instruments, so maybe that is what you want, yes. But with only n=N=2 instruments, considering superpopulations, ie, the mechanism involved, though I have not worked with ICCs, I don't think most statistics would be very stable. I would think that someone else reading this would know something about this. I have no other ideas/suggestions, and am not an ICC proponent.

We keep going with the same subject...

When ICC was proposed, its assumed that the measurements are independent or it's necessary to be? (to apply correctly the icc)

What is your opinion about this?

Dear Paulo

In my opinion, dependent data may yield conflicting results in terms of ICC, but you should consult a statistician.

Have a look at the attached paper, from Müller and Büttner.

It provides an interesting review of the ICC, the underlying assumptions we have discussed (normality of distributions, equality of variances), the kind of ICC to choose, the severe problem of the range of the measuring scale (the wider, the better the result), the Bland-Altman plot as an alternative and the statement that assessing agreement by solely computing an ICC must be regarded as insufficient.

Kind regards

Emilio

Hello Emilio, thanks a lot for your contribution...

If I use ICC in dependente data, what are the consequences? (any) Error?

Conceptually, I prefer calculating ICC from independent measurements, but I cannot explain to what extent the results may be biased if you use a dataset including dependent data.

I haven't found any reference about that, but surely there are people in RG who can give you an answer.

Regards

Also haven't found any reference about that!! Hope someone can help, because I need to know if I apply some techiniques, not assuming data indepedence, it will affect what? My decision?