Hi,
I am trying to find a way to express the importance/relevance of a cross-level interaction in a hierarchical linear model (so not in terms of significance). I have a level 2 moderator M (standard normal) influencing the relation between X and Y (both level 1).
Can I express the relevance in terms of the variability of the slope of the level 1 variable?
Let me give you an example: my cross-level interaction is .10. Let's say that for someone with an average value on M, the slope is .15, so for someone with a high score on M (+1), the slope is .25.
The variance of the random slope of the level 1 variable is .03 (hence the SD of the slope is sqrt(.03) = .17). Intuitively, I would say that the effect of the moderator M is quite strong, because it amounts to (.10/.17 =) .59SD of the random slope of X.
I don't think I have seen this being done anywere, but conceptually it makes sense I think. Any thoughts?
Best,
Dirk
I would not do this - the whole point of of cross-level interaction to account for the variation in the random slopes - an important cross-level interaction would reduce the unexplained variance of the slopes.
To interpret the size of the effects I make and graph conditional predictions plotting the predicted response for characteristic values of the two predictors involved in the interaction for characteristic values (eg percentiles) keeping all other predictors at their average value . For an example see Figures 2 and 3 of
https://www.researchgate.net/publication/232249281_An_exploratory_multilevel_analysis_of_income_income_inequality_and_self-rated_health_of_the_elderly_in_China
This gives a strong impression of the comparative size of effects
This procedure is enabled in MLwiN through customized predictions
https://www.researchgate.net/publication/234015221_Manual_supplement_for_MLwiN_Version_214
Article An exploratory multilevel analysis of income, income inequal...
Data Manual supplement for MLwiN Version 2.14
I agree. The variance of the random slop in a model that contains the cross level interaction is conceptually the variance of the slopes that is not accounted for by the Level-2 predictor (i.e. your residual variance of the slope).
What you might do (in addition to the graphical inspection) is run your model with all predictors except the cross-level interaction and look at the random slope variance there. This variance is the total random slope variance (the random slope in this model is not predicted by any variable). Let's assume this varaince is .25 in your example. If you now include your cross-level interaction, the random slope variance should be reduced (because you explain part of your variance by the level 2 predictor) and is now .17. You have reduced the variance by 32% (.17-.25)/.25. This procedure is based on Pseudo R² as suggested by Xu (2003):
http://onlinelibrary.wiley.com/doi/10.1002/sim.1572/abstract
There are some issues with this estimate (see https://www.researchgate.net/publication/275442223_Explained_Variance_Measures_for_Multilevel_Models) but it is intuitive and rather straightforward. I am also not aware of any better alternative - I'm happy to be told otherwise though.
Hope this helps. Best,
Andreas
Article Explained Variance Measures for Multilevel Models
Dear Kelvyn and Andreas,
thank you both for your answers. I actually did both approaches but was looking for a more discrete "high" or "low" effect size criterium,
For example, in the approach described by Andreas, the inclusion of the cross level interaction resulted in a decrease of 4% in random slope variance. But what does this say? Is 4% a lot? Or is it little? I know this is not any different from intepreting R2 values in "normal" OLS but I was looking for a more clear benchmark.
The same holds for the procedure described by Kelvyn. If I understand your answer correctly, that is what I described in my original question: for an average value on M, the slope is .15, so for someone with a high score on M (+1 SD), the slope is .25, for someone with a low score (-1 SD) the slope is .05. Again, I would ask: is this difference of .20 large or small?
Using unstandardized variables, whether .20 is large or small of course depends on the scale of Y. That is why I am trying to reference it back to the variability/range of the Y variable to get a better idea of the relevance of effect size.
Dirk
you may want to look a this current exchange
https://www.researchgate.net/post/Multilevel_Modeling_standardized_beta-coefficients_Cohens_r_effect_size
I think it has to be done by the substantive subject specialist (you?) by reflecting on how much of a difference there is for Y for different values of the Xs. Is it millimetres or meters or kilometers etc?
Kelvyn
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Structural Equation Modeling