In our study, each participant rated several organizations randomly selected from a big pool on some attributes (denoted as) x1, x2, and x3 and an outcome y, and each organization has multiple raters. Moreover, we also collected personality data m1, m2, and m3, of participants. Because of this data structure, we used a multilevel mixed-effects model.
We have calculated group means of x1, x2, and x3 and then group-mean ("group" here means organization) centered x1, x2, and x3. We can denote the group means of x1, x2, and x3 as T1, T2, and T3 ("T" represents level two). So T1, T2, and T3 are level-2 analougies of level-1 x1, x2, and x3. If I understand it correctly, when simultaneously including T1-T3 and x1-x3 as predictors (we included them all because we wanted to investigate the "predictive" power of each predictor while controlling the rest), the coefficients of x1-x3 represent the level-1 associations between the predictors and the outcome and the coefficients of T1-T3 represent the level-2 associations.
But what to do when we want to investigate the interactive effects between the predictors and the outcome? We prefer to use level-2 T1-T3 instead of level-1 x1-x3. The advantage of level-2 T1-T3 is that they are aggregated through multiple raters and should be more reliable and "objective". So I can add an interaction term, for example, between T1 and m1, and it represents the cross-level interaction effect of T1 and m1 on the outcome.
My questions are:
1. should I exclude all level-1 x1-x3 predictors when investigating cross-level interaction effects?
2. Should I include a corresponding interaction term between level 1 x1 and m1 to do some "control"?
Thank you for your answers!
We have around 400 organizations and 800 participants. The materials are real company descriptions, and they were randomly selected from the real world (extracted from websites). Each participant rated 6 randomly selected orgs from the 400 and each org has 12 random raters. We included random intercepts for both participant and org. Because of model convergence issues, we didn't include random slopes. I hope this information can help!
It is non-hierarchical. Level 1 is the trial (so the rating and if you have things like response time), but participant and org are both at a higher level, to the level 1 v level 2 terms can get confusing. Further, as it is multivariate, som use an extra level to describe this. In short, it is a fairly complex model so convergence may be an issue because of this (but try some of the things in https://rstudio-pubs-static.s3.amazonaws.com/33653_57fc7b8e5d484c909b615d8633c01d51.html). Everything cannot be included in the model (remember, models aren't supposed to be right, they are supposed to aid wise decision making). Were simpler models converging? If so, make sure you use those for starting values.
Daniel Wright Thank you for your answer! I totally agree that there is no hierarchical structure with all level-1 predictors. Because each organization has 12 raters, it is possible that we can calculate the average scores of organizations (level-1 predictors) from their 12 raters. If I understand it correctly, these average scores are level-2 analogies of the level-1 predictors. When we include both level-1 predictors (after group-centered) and their level-2 analogies (average scores), the model becomes a multilevel model. A simplified reason for doing this is that we assume that all raters have their own limitations (rating errors) that lead to rater disagreement (again, this is a simplified reason, and it is more complicated). So we believe that by averaging scores from different raters can get more "objective" ratings as predictors.
I know that some studies in the organizational science field do similar analysis to research the so-called "emergence phenomena" where associations between level-1 predictors and the outcome differ from associations between level-2 analogies and the outcome. But I'm not sure what to do in our case, whether we need to include all the level-1 predictors and level-2 analogies at the same time when investigating the main effects of the predictors (but we may only interpret the coefficients of the level-2 analogies because we believe they are more reliable) and when investigating the interaction effects between level-1 personality traits and level-2 analogies.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Meta-Analysis