Hi everyone, I'm having some questions about choosing the right test for my experiment and I hope I can find help here!
Here are the settings of the experiment:
1. participants are randomly assigned to either the experimental or control group.
2. then both groups undergo two sessions of treatment (either the experimental treatment or the control treatment).
3. Outcome variables (all continuous) are measured twice: after session 1 and after session 2.
My research questions:
RQ1: whether the outcomes are better for the experimental group than the control groups? (so the main effect of groups)
RQ2: whether there are changes over the two sessions, and whether the change is different for both groups? (so the main effect of time and the interaction between time and group)
I first opted for a mixed ANOVA since this is a simple between-within subject design. But I also read that multilevel/mixed modeling is generally preferred. I don't see what can be a random effect in this design as well (please correct me if there is a random effect!)
Hence, which one should I choose for my work?
Thanks a lot!!
You are right that in general, multilevel approaches are superior to ANOVAs. Assuming that you have multiple observations per participant and all participants answer all items participants are nested in items and vice versa, which leaves you with a multilevel structure.
You could use both, items and participants as random intercepts, but you could also assume differential effects of your IVs on your participants.
A linear mixed model (or multilevel model) would be advantageous if for example participants are nested within therapists (or who else delivers the treatment). Then, a multilevel model could consider this dependency in the data (participants treated by one therapist are more similar to each other than participants treated by different therapists).
It is true that the multilevel model could additionally model differences between participants by using them as random effect. However, since there are only two measurements nested within participants, the reliability of this approach is questionable. Have a look at the literature on sample size issues in multilevel modeling. The ANOVA considers differences between participants by analyzing only intraindividual changes and excluding between-participant differences in the within-effects.
In your case, since you only want to compare the groups, I think both models should do the trick.
You can assess the need for a multilevel model. I quote from Field et al. (Discovering statistics using R, p. 878): If there is not significant variation across contexts in the first place, then doing multilevel modeling is simply a perverse exercise in mental flagellation. To test the need, compare the model with the intercept only with a model with the random intercept for the context.
I think you should stick to the mixed ANOVA.
I agree that I would also stick with the mixed ANOVA in this situation. However, many authors who published on multilevel models won't agree with the statement that you don't need to consider the multilevel structure when there is no significant intra-class correlation (ICC). Even if the ICC is not significant (i.e., the null model with the random intercept does not fit the data significantly better than a linear model without random intercept) you should consider the multilevel data structure, because even in the absence of a significant ICC the standard error could be biased due to the nesting leading to a significance test that is too liberal.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA