Hi everyone
Hope you can help me with this, as I am currently working on a meta-analysis I did in a smaller scale as part of my Phd dissertation, which is now becoming a paper to be submitted. However, no one in my department is experienced with meta-analysis, and I am therefore somewhat alone in making decisions on how to do analyses etc.
I chose to do a three-level "multilevel" analysis as I have several effects sizes from the different studies (in total 131 effect sizes from 20 studies). However, when I compared a two-level model with a three-level model, AIC and BIC indicated a better fit for the more simple, two-leveled model.
However, when reading the literature, it is suggested to keep a three-level model still, if it is based on solid theoretical rationale:
"However, please note that there are often good reasons to stick with a three-level structure–even when it does not provide a significantly better fit. In particular, it makes sense to keep a three-level model when we think that it is based on a solid theoretical rationale. When our data contains studies with multiple effect sizes, for example, we know that these effects can not be independent. It thus makes sense to keep the nested model, since it more adequately represents how the data were “generated" (from Harrer "doing meta-analysis in R").
I still find it difficult to argue for a three-level model when data says otherwise. What are some of the "solid theoretical" arguments to consider, that would weigh towards going with either model? If I could go with a less complex model, this would of course be preferred, but would a reviewer questions the decision if the model comparison backs up a two level model?
All the best, and thank you
Katrine
I would second the argument of keeping the model which makes greater sense for your study field and biological expertise (in that case the 3 level model). Do not forget that your aim in the stat analysis was not to prove that a 2 level model was better than a 3 level model. In various studies, the hierarchical structure of the data is defined a priori and should be the default if it makes biological sense. You can go back to a simpler model if there are some convergence issue or any other singularity problem.
What your analysis tells you is that BIC or AIC chose the simpler model if the fit is not better after adding 1 specific new explanatory model. It does not test for biological plausability of 1 model vs another.
My 2 cents and good luck for your study and PhD!
You might also want to look more closely at the outcome measures of the studies. Are these expressions of more or less the same thing? Does it make sense to lump them all together? Maybe your model fits better if you do separate analyses for different categories of outcome measures. And this could in some instances also be more informative.
When conducting a meta-analysis, choosing between a two-level and three-level model depends on the structure of the data and the research question being addressed.
A two-level model is appropriate when the studies included in the meta-analysis have only one level of data, meaning they provide information on the effect size (e.g., mean difference, correlation) and some sample-level characteristics (e.g., sample size, standard deviation). In this case, the studies are treated as the first level and the sample-level characteristics are treated as the second level in the analysis.
On the other hand, a three-level model is appropriate when the studies included in the meta-analysis have multiple levels of data, such as studies that include data from different subgroups or studies that have been conducted in different contexts (e.g., different countries, different time periods). In this case, the studies are treated as the first level, the subgroups or contexts are treated as the second level, and the sample-level characteristics are treated as the third level in the analysis.
In summary, a two-level model is appropriate when the studies provide information on the effect size and some sample-level characteristics, while a three-level model is appropriate when the studies have multiple levels of data, such as different subgroups or contexts. The choice between the two models ultimately depends on the structure of the data and the research question being addressed in the meta-analysis.
I think Ma'Mon Abu Hammad is right about this, but it does not have to be different subgroups for three-level to be relevant. The three-level analysis can also help deal with the dependency of different outcomes from the same population. Traditionally, meta-analyses have needed all the effect sizes to be independent. Dependency of effect sizes represents an overlap in information and could lead to overconfidence in the meta-analytic results. Thus, to achieve independence, typically only one effect size was included from each study. This, in turn, is questionable for other reasons, as extracting one effect size from each study assumes homogeneity of effect sizes within studies. By applying a three-level model to meta-analysis, the biases associated with independence can be mitigated as one can allow effect to vary between participants (level 1), outcomes (level 2), and studies (level 3).
Still, as I tried to say in my previous comment, not all outcomes should be grouped together. Some "outcomes" can be more like a mediator or secondary to the goal of therapy (e.g., therapeutic alliance, drop-out rates, acceptability of the intervention) while others measure level of desired endpoint state (symptoms, quality of life). It may also be differences in proximity or alignment to the intervention that makes it less relevant to lump them together. Effect size is basically signal/noise-ratio, measuring adherence to a specific skill the therapist is teaching the patient is likely to have a larger effect than the therapy's impact on symptom levels, marital happiness or income seven years later.
I found this article helpful both for the rationale and method:
Assink, M., & Wibbelink, C. J. (2016). Fitting three-level meta-analytic models in R: A step-by-step tutorial. The Quantitative Methods for Psychology, 12(3), 154–174.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Probability
- Probability Theory
- Reliability