Hi everyone
Hope you can help me with this, as I am currently working on a meta-analysis I did in a smaller scale as part of my Phd dissertation, which is now becoming a paper to be submitted. However, no one in my department is experienced with meta-analysis, and I am therefore somewhat alone in making decisions on how to do analyses etc.
I chose to do a three-level "multilevel" analysis as I have several effects sizes from the different studies (in total 131 effect sizes from 20 studies). However, when I compared a two-level model with a three-level model, AIC and BIC indicated a better fit for the more simple, two-leveled model.
However, when reading the literature, it is suggested to keep a three-level model still, if it is based on solid theoretical rationale:
"However, please note that there are often good reasons to stick with a three-level structure–even when it does not provide a significantly better fit. In particular, it makes sense to keep a three-level model when we think that it is based on a solid theoretical rationale. When our data contains studies with multiple effect sizes, for example, we know that these effects can not be independent. It thus makes sense to keep the nested model, since it more adequately represents how the data were “generated" (from Harrer "doing meta-analysis in R").
I still find it difficult to argue for a three-level model when data says otherwise. What are some of the "solid theoretical" arguments to consider, that would weigh towards going with either model? If I could go with a less complex model, this would of course be preferred, but would a reviewer questions the decision if the model comparison backs up a two level model?
All the best, and thank you
Katrine