We are conducting a meta-analysis to scrutinize the toxicity of a compound across a set of published studies. In some studies, toxicity is reported for a single test species, in others - in several species. Moreover, in some studies, several concentrations are being tested as well as different test conditions.
It is well known that to keep observations independent, only a single entry per study can be used. However, it feels acceptable to use one entry per species. Can this be justified?
Another question is how (or whether it is possible at all?) to treat different concentrations from the same study in, for example, metaregression? To set up the data for such regression, one has to use toxicity at each concentration as a separate entry...
Of course, the most like answer is that no multiple entries are allowed, full stop... but it is very tempting to use the data:)
Thank you/Elena
There is nothing wrong analyzing more than one outcome (aka dependent variable). It all depends on your research question (or questions). For example, I could see a question like "Compared to control, what is the effect of the compound on Species A". You could have another question like "Compared to control, what is the effect of the compound on Species B". In this situation, you can treat it like you did two separate systematic reviews/meta-analyses and you might have a different number of studies in each.
Thank you, Gordon! If I understand this correctly, you are referring to two separate models, one for species A and another for species B (both are coming from the same study). I am afraid that my question is about using species A and B in the same model with a question: "Compared to control, what is the effect of the compound on Species A, B, C and D":)
For example, I want to evaluate effects of compound X on mussels. Study "Name, Year" reports a non-significant positive effect in Mussel A and non-significant negative effect in Mussel B. Choosing only one species for the metaanalysis is difficult and can potentially affect the overall effect. My question is: can I use both as entries in the same model named as "Name, Year (1)" and "Name, Year (2)"?
/Elena
Dear Elena,
Non-independence is a common issue in meta-analyses (especially in biology and ecology).
In R (e.g. in the ‘metafor’ R package) you can fit multilevel models accounting for different studies using same species using random (non-)nested structure, e.g.: ‘(1|study/observation.id) + (1|species)’ (see e.g. Midolo et al., 2019). N.B. the “observation.id” within the “study” grouping level in the random structure of the models to account for the possibility that the underlying true effects within each study or experiment are not homogeneous (see Konstantopoulos 2011).
In case you have multiple effect sizes per species and studies in multi-group experiments - e.g. different groups of individuals treated with different doses of compound X and compared to a single ‘control group’ (see figure 3C in Nakagawa et al. 2017) – then, you can probably account for correlated effect sizes by imputing the covariance matrix to be used in the models (see Lajeunesse et al. 2011) (assuming you are using contrast-based effect sizes such as response ratios).
I recommend to have a look at these two papers: Nakagawa & Santos (2012); Nakagawa et al (2017) addressing some related issues and proposing general solutions for other problems you might encounter (e.g. test of funnel plot asymmetry in non-independent meta-analyses).
---
Konstantopoulos, S. (2011). Fixed effects and variance components estimation in three‐level meta‐analysis. Research Synthesis Methods, 2(1), 61-76.
Lajeunesse, M. J. (2011). On the meta‐analysis of response ratios for studies with correlated and multi‐group designs. Ecology, 92(11), 2049-2055.
Midolo, G., De Frenne, P., Hölzel, N., & Wellstein, C. (2019). Global patterns of intraspecific leaf trait responses to elevation. Global change biology, 25(7), 2485-2498.
Nakagawa, S., Noble, D. W., Senior, A. M., & Lagisz, M. (2017). Meta-evaluation of meta-analysis: ten appraisal questions for biologists. BMC biology, 15(1), 1-14.
Nakagawa, S., & Santos, E. S. (2012). Methodological issues and advances in biological meta-analysis. Evolutionary Ecology, 26(5), 1253-1274.
Excellent answer, thank you, Gabriele! I will go throught these papers but is it so that multilevel model can be used when species are not overlapping between studies? For example, when study 1 has species A and B and study 2 has C and D it would still be possible to run a model that would anclude all 4 species?
/Elena
It's certainly acceptable. For ex if you have 10 studies in the meta-analysis and one of them using two different concentration/drug groups compared to one control. You can treat each study group as a separate study compared with the control. So when you come to the effect size estimation you will have 10 studies and 11 comparisons. Below is a practical example hopefully will answer all your questions. BTW, congrats for the good thinking
https://www.ncbi.nlm.nih.gov/pubmed/?term=Transversus+Abdominis+Plane+Block+to+Ameliorate+Postoperative+Pain+Outcomes+After+Laparoscopic+Surgery%3A+A+Meta-Analysis+of+Randomized+Controlled+Trials
You are welcome, Elena – hope this is useful for your research.
I think it is fine to model different species together with multilevel models, if needed. In general, I would say that using multilevel models makes sense in case multiple estimates of the effect size are nested within a higher-level grouping variable (normally this is study or experiment id) – so, this choice is not related on having the same/different species within or across studies.
Dear Hytham,
Thank you very much for the paper. This is what we were doing, in principle, but then it is a violation of the independence assumption... When using the outcomes of the same study as independent observations, it is possible that there was an effect of the observer (research team, design, accuracy, source of populations, etc.) on the outcome. Do you know if there is any reasoning behind the acceptance of such multiple outcomes?
/Elena
Honestly I don't know any reasoning but it's a good option to make use of the data especially if the number of available studies is small, like in the paper I gave as an example. We recently did a Meta-analysis (under review) and we did the same as we had only 10 RCTs.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Meta-Analysis