I am assisting a student with a measure of agreement for test-retest data using an ICC (2, 1) for absolute agreement. The data is ordinal with a 5 point scale, and according to Streiner et al 2015's book, the ICC is superior in these situations than (for example) kappa or weighted kappa.
We've encountered a situation where despite nearly all scores being identical between test 1 and test 2, we have an ICC of .000. As far as I can tell this is because there is almost no variation in the scores. For both test 1 and test 2, nearly all of the answers given are "2". I've queried with the student whether this suggests that there was something wrong with the question in the first instance, since if you ask a question and everyone answers identically, was it really worth asking in the first place?
However what I'd like to know is WHY the ICC is 0 despite nearly all responses agreeing. My understanding is that ICC is a version of correlation adjusted for the fact that the different variables are measuring the same thing. Correlation looks at associations between two variables. If when plotted on a scatter plot scores on both variables are, with very few exceptions, all "2", then it is not possible to model a relationship between the two variables. They're not associated with one another, they're functionally identical.
Is this correct? Thank you for any help you can provide.
Dear Gavin,
The problem could refer to the following:
1- The scale that was used to collect the data was neither reliable nor valid
2- The data used was not normally distributed.
3- the size of the sample was very low.
Regards
Dear Gavin,
please provide us with some more information. How large was your sample? May it be that at least one test has no variance, i.e. really all answers are 2 (not only nearly all)?
If the range of test scores are restricted, then the ICC will be forced to be low. Recall that the ICC compares the within-subject variability to the between-subject variability (or total variability). If the range of test scores is severely restricted, then much or most of the variability will reside between test scores within subjects. This is a common design mistake in selecting subjects for assessment. To assess ICC, subjects should be selected across the full range of possible scores, not just in a restricted area of the range.
Paul
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics