I am assisting a student with a measure of agreement for test-retest data using an ICC (2, 1) for absolute agreement. The data is ordinal with a 5 point scale, and according to Streiner et al 2015's book, the ICC is superior in these situations than (for example) kappa or weighted kappa.
We've encountered a situation where despite nearly all scores being identical between test 1 and test 2, we have an ICC of .000. As far as I can tell this is because there is almost no variation in the scores. For both test 1 and test 2, nearly all of the answers given are "2". I've queried with the student whether this suggests that there was something wrong with the question in the first instance, since if you ask a question and everyone answers identically, was it really worth asking in the first place?
However what I'd like to know is WHY the ICC is 0 despite nearly all responses agreeing. My understanding is that ICC is a version of correlation adjusted for the fact that the different variables are measuring the same thing. Correlation looks at associations between two variables. If when plotted on a scatter plot scores on both variables are, with very few exceptions, all "2", then it is not possible to model a relationship between the two variables. They're not associated with one another, they're functionally identical.
Is this correct? Thank you for any help you can provide.
Dear Gavin,
The problem could refer to the following:
1- The scale that was used to collect the data was neither reliable nor valid
2- The data used was not normally distributed.
3- the size of the sample was very low.
Regards
Dear Gavin,
please provide us with some more information. How large was your sample? May it be that at least one test has no variance, i.e. really all answers are 2 (not only nearly all)?
If the range of test scores are restricted, then the ICC will be forced to be low. Recall that the ICC compares the within-subject variability to the between-subject variability (or total variability). If the range of test scores is severely restricted, then much or most of the variability will reside between test scores within subjects. This is a common design mistake in selecting subjects for assessment. To assess ICC, subjects should be selected across the full range of possible scores, not just in a restricted area of the range.
Paul
That's the sort of answers I was expecting, thanks. After thinking about it for a while longer I also realized that the ICC measures reliability, with agreement being secondary to this. These two things are to an extent independent. In the case of the data we have here we have high agreement but poor reliability. If you're not doing an ICC for "absolute" agreement it is also possible to have the reverse be true - a situation where you have high reliability but low total agreement.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics