For a multilevel analysis, we need to aggregate a lower level variable to a higher level variable. What is the cutoff point for Intraclass correlation (ICC(1) and ICC(2)? Is there any agreement for the cutoff point?
LeBreton and Senter (2008) have suggested that an ICC(1)=.05 represents a small to medium effect (p. 838), Bliese (1998) has simulated conditions where only 1% of the variance is attributed to group membership ICC(1)=.01) and, still, strong group-level relationships were detected that were not evident in the lower level data.
For assessing reliability of group-level means, ICC(2) 0.75 are excellent (Fleiss, 1986).
References:
Bliese, P. D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1 , 355–373.
Fleiss, J. (1986). The Design and Analysis of Clinical Experiments. Wiley, New York.
James, D. L., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. The Journal of Applied Psychology, 69, 85–98 LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11 , 815–852
THis paper (sadly not yet complete!) discusses some of the issues especially in relation to measurement error of aggregated variables - I know of no rule of thumb;educational studies typically have 10 percent of the variation at the higher level and use aggregate variables
Paper is on RGate:
Do multilevel models ever give different results?.
https://www.researchgate.net/publication/252146040_Do_multilevel_models_ever_give_different_results?ev=prf_pub
Article Do multilevel models ever give different results?.
Yes. Intraclass correlation. ICC(1) and ICC(2) varies a lot depending on the group size (the number of people who assess higher level variables). That is, if we have a small group, it is very difficult to get relatively good numbers. Any justification for this situation?
If you have a look at the paper
Do multilevel models ever give different results?.
https://www.researchgate.net/publication/252146040_Do_multilevel_models_ever_give_different_results?ev=prf_pub
It shows one way to ameliorate this problem - that is use another multilevel model with level 1 explanatory variable as the response variable and use the model to estimate precision-weighted estimates at level 2 which are then used with the original response
The importance of using such ' contextual' or aggregate level 2 variables is considered here
https://www.researchgate.net/publication/233756428_Explaining_Fixed_Effects_Random_Effects_modelling_of_Time-Series_Cross-Sectional_and_Panel_Data
Article Do multilevel models ever give different results?.
Article Explaining Fixed Effects: Random Effects Modeling of Time-Se...
Amir Riaz, could I bother you with the question as to where you got these threshold values?
I would like to provide some help here, but I confess to not understanding the original question fully. I believe that by intraclass correlations, reference is being made to intraclass correlation coefficients (ICCs) - the statistic that is used to determine the amount of consistency / agreement between sets of scores / data.
However, from that point I am lost. ICCs are often identified by two integers within parentheses - e.g., ICC(1,1) or ICC(2,1) where the first integer refers to the model of the ICC and the second integer refers to the form of the ICC. Furthermore, ICCs need to be identified in terms of their type (consistency or absolute) in addition to their model and form. It is important to be clear about what model, form, and type of ICC is being referred to at any point in time because there are 10 different kinds of ICC depending on those three variables.
So when someone refers to something like ICC(1), I am unsure what is being referred to.
If someone could set me on the right path regarding this, I might be able to offer additional help. In the interim, I would recommend a cutoff of at least .70 before agreement between variables could be considered "respectable", and at least .90 for clinical situations.
I tentatively suggest that an article that I wrote recently might be helpful. The reference is:
Trevethan, R. Intraclass correlation coefficients: clearing the air, extending some cautions, and making some requests. Health Services Outcomes and Research Methodology. doi:10.1007/s10742-016-0156-6
This article has not been allocated a volume and page numbers yet, but it is available online.
In it, I refer to different cutoffs for ICCs and provide references that indicate different sets of cutoffs as being suitable depending on particular contexts.
I would be happy to provide more advice if someone could clarify the original question for me.
Robert
LeBreton and Senter (2008) have suggested that an ICC(1)=.05 represents a small to medium effect (p. 838), Bliese (1998) has simulated conditions where only 1% of the variance is attributed to group membership ICC(1)=.01) and, still, strong group-level relationships were detected that were not evident in the lower level data.
For assessing reliability of group-level means, ICC(2) 0.75 are excellent (Fleiss, 1986).
References:
Bliese, P. D. (1998). Group size, ICC values, and group-level correlations: A simulation. Organizational Research Methods, 1 , 355–373.
Fleiss, J. (1986). The Design and Analysis of Clinical Experiments. Wiley, New York.
James, D. L., Demaree, R. G., & Wolf, G. (1984). Estimating within-group interrater reliability with and without response bias. The Journal of Applied Psychology, 69, 85–98 LeBreton, J. M., & Senter, J. L. (2008). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods, 11 , 815–852
I imagine Amir referred to this paper Maurits van Leeuwen
Article Choosing the best index for the average score intraclass cor...
But, the values are median values that the authors report for the ICC computations.
The paper gives a good idea about the interpretation of the different ICCs.
- Badges
- Science topic