There are a lot of issues one could discuss here but the factor I think is most important in affecting the quality of your meta-analysis is the quality of your systematic review. Your meta-analysis can only be as good as the data that goes into it and that is dictated by the quality of your systematic review. Publication bias will always be an issue one will have to deal with. Meta-analyses of interventional studies will always be plagued with the fact that nonsignificant studies and studies with negative findings will be less likely to be published and thus make it into your meta-analysis. The objective of a systematic review is to find ALL studies that meet the inclusion/exclusion criteria for a given research question. And that means finding the unpublished studies as well. Short of finding all studies, one should at least find a random sample of ALL studies (and how one would verify that, I don't know). My personal opinion is that most people conducting systematic reviews don't try hard enough to find ALL studies.

Chapter 32 in Rothman and Greenland 2nd ed 1998 p 643-673 is a very useful comprehensive coverage of methods and disadvantages/pitfalls of meta analysis.

The attached paper by Ioannadis et al is also v useful.

The meta-analysis can potentially identify trends in the available data not detected in the original research. It is likely that a better estimate of the true population effect, as well as potential moderators, can be determined by using this aggregate level data due to the increased statistical power. However, a systematic review and meta-analysis can only be as good as the articles on which it is based.

I agree with Dr. Warren that the most important issue impacting the quality of your systematic review and meta-analysis is identifying all of the relevant articles on your topic of interest. Typically, we recommend searching a minimum of 3 electronic databases, manually searching the references of previous systematic reviews and of the original research articles returned during the electronic database search, and contacting authors for missing or incomplete data (as well as other potential articles not discovered during the database or manual search).

Although there may be no way of knowing if you truly have identified all of the relevant articles on a given topic, the electronic search should identify 90% of the articles in your systematic review. If you find that the records returned in your manual search are accounting for a greater percentage, your keywords for the electronic search should most likely be re-evaluated and the search should be performed again.

These articles may help with the systematic review:

Whiting P, Westwood M, Burke M, et al. Systematic reviews of test accuracy should search a range of databases to identify primary studies. J Clin Epidemiol. 2008;61(4):357–64.

Papaioannou D, Sutton A, Carroll C, et al. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010;27(2):114–22.

I agree with the previous comments on data. Most of the meta-analysis papers I've read suffer from either not having enough studies included, or not performing a sub-group analysis. It seems that a lot of researchers want to base their conclusions solely on the overall effect, without addressing how moderator variables create heterogeneity in the effect, which gives a much more comprehensive explanation of the research.

i) Garbage in, garbage out. Disadvantage of a meta analysis is that it puts your garbage in fancy looking package and can fool both you and your readers.

ii) You can make a bad cake with good onions (just made that up). There may be nothing wrong with the research but the assumption of the meta analysis is that these all belong together in some way. That can be (and has been) stretched to the point of absurdity in some cases. Even sub-group analysis does not fully mitigate this in that it assumes a common effect is possible with sub group variation. I have seen a meta analysis of the effect of 'research based nursing interventions' - utterly absurd!

One cannot overemphasize enough the potential, if not almost certain of the effect publication bias eluded to in Gordon's and others' responses. Studies with positive outcomes are significantly more likely to be published and this most certainly constitutes a clear one-sided approach. Additionally, the impact of the systematic review process itself ought to be appropriately accounted for in the interpretaion oof the utput of the meta-analysis ; something that is not always done and misleading results are less than a step away when one or more guiding principles of this methodology are violated. Also, some research questions would inherently or not be more prone to these biases than others.

I believe, none!

Meta-analysis is considered as the strongest clinical evidence. Though driving an excellent car doesn't mean you are a good driver. The interpretation and handing of meta-analysis software properly is very important. Meta-analysis can be awful when researchers would cheat/hide their limitations and interpret the way they wanted to visualize. Though I haven't come across any that kind of meta-analysis, however, there is a possibility. So, good luck!

Meta-analysis is a statistical analysis that combines the results of two or more independent studies, generating a single estimate of effect. Meta-analysis estimates with more power and precision the "true" effect size of the intervention, often not demonstrated in single studies, with inadequate methodology and insufficient sample size. A systematic review does not necessarily need to present a meta-analysis. On the contrary, in some cases it is not appropriate to be performed, and may even lead to erroneous conclusions (Cook, Mulrow, Haynes, 1997, O'Rouke and Detsky, 1989). For example, when only two studies reported data from one outcome with estimates of opposite effects, i.e., one study points out benefit of the intervention and the other indicates benefit of control. In this case, these findings should not be combined.

Meta-analyzes combine data from all studies evaluating the same or similar treatments for patients with a specific condition. In this way, effect estimates are obtained with greater precision (decrease of the random error) due to the increase of the sample size. Meta-analysis can also contribute to increase the external validity of the studies, that is, to contribute to the generalization of the results. The findings of a particular study may be valid only for a population with the same characteristics as those investigated in the study (Egger and Smith, 1997).

However, meta-analysis does not always conclude or define the true effect of the intervention. If only a few studies with small sample sizes or low number of events are included, the effect estimate generated by the meta-analysis will present a large confidence interval, which may include the value "1". The value 1, which represents no difference between treatments or no statistical significance. In cases where the confidence interval is very broad, it can be understood from a significant benefit in favor of the intervention to an increase in the risk attributed to it, it can not be concluded that there are no differences between the treatments. In these situations, the meta-analysis will be inconclusive and may suggest the need for studies with greater statistical power to conclude the question (Egger and Smith 1997; Gelber and Goldhirsch 1987).

It should be noted that systematic reviews of RCTs are observational and retrospective studies. Thus, the results of a meta-analysis should be interpreted considering the limitations and biases inherent in observational studies (Mulrow & Cook; Davidoff, 1997). The main limitations that can affect systematic reviews are publication bias (and others similar as language bias), bias risks in primary studies (methodological limitation of primary studies), and difficulties in combining studies that may have differences in populations, interventions, comparators and definition of outcomes (clinical heterogeneity). Thus, it is necessary to critically explore the evidence presented by meta-analysis. Misleading analyzes can be avoided if some basic principles in conducting the systematic review are observed / followed.

The search strategy to identify relevant studies should be clearly delineated. In particular, it should be determined how much the search strategy will be extended to include unpublished studies, since its results can systematically differ from published studies. Meta-analyzes restricted to published evidence may produce distorted results due to publication bias. For location of published studies, electronic databases are essential and useful, but if they are only used, a substantial proportion of relevant studies may not be retrieved. It is noteworthy that many studies are finalized, but are not published. The main causes of this association would be non-significant or negative results. The identification of these studies is not easy. One possible way is by searching specific databases that record study protocols that will be conducted. For instance, the Clinical Trials () (Dickersin & Sherer, Lefebvre, 1994; Higgins, 2011).

References

Higgins, J. P. (2011). Green S. Cochrane handbook for systematic reviews of interventions version 5.1. 0. The cochrane collaboration, 5(0).

Cook, D. J.; Sackett, D. L.; Spitzer, W. O. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam Consultation on metaanalysis. Journal of Clinical Epidemiology, Maryland Heights, v. 48, n. 1, p. 167-171, 1995.

O’Rourke, K.; Detsky, A. S. Meta-analysis in medical research: Strong encouragement for higher quality in individual research efforts. Journal of Clinical Epidemiology, Maryland Heights, v. 42, n. 10, p. 1021-1024, 1989.

Egger, M.; Smith, G. D. Meta-Analysis: potentials and promise. British Medical Journal, London, v. 315, n. 7119, p. 1371-1374, 1997.

Gelber, R. D.; Goldhirsch, A. Interpretation of results from subset analyses within overviews of randomized clinical trials. Statistic in Medicine, Malden, v. 6, n. 3, p. 371-388, 1987.

Dickersin, K.; Scherer, R.; Lefebvre, C. Identifying relevant studies for systematic reviews. British Medical Journal, London, v. 309, n. 6964, p. 1286-1291, 1994.

In addition, if the systematic review includes meta-analysis, then a meta-analysis should be calculated for each outcome. However, the combination of the results is not simply the sum of the populations of the studies, as if they were all part of the same large study. First, a statistical summary is calculated for each study and then the weighted average of the effect estimates of the included studies to calculate the meta-analysis is considered. Sample size and number of events are the determinants of study weights in the meta-analysis. Sample size and number of events are the determinants of study weights in the meta-analysis. The relationship between weight and sample size is complex and will depend on the statistical method chosen to perform the meta-analysis. The statistical summary of the treatment effect for each study is calculated from the identification of the nature of the data and definition of the effect estimate. In parallel, the analysis model of the meta-analysis is defined and the statistical method applicable to the model type.

Once the nature of the data is identified and the effect measure to be calculated is established, the analysis model of the meta-analysis is defined. The options are fixed-effect model and random-effect model. Numerous statistical methods are used to weight effect estimates among the studies included in the review. The most commonly used methods in systematic reviews are (Inverse variance method / Mantel-Haenszel method / Peto method).

Like clinical trials, systematic reviews of RCTs are not free of biases that may compromise the quality of the evidence generated. They are introduced in various ways in the process of location and selecting studies. It is important that these biases are identified in order to critically examine the process of conducting the review, as they may compromise the results of the meta-analyzes. The 7 main types of systematic reviews biases are: 1) publication bias; 2) time bias; 3) multiple publication bias; 4) location bias; 5) citation bias; 6) language bias; 7) outcome bias.

An important component of the systematic review is to investigate the consistency of the treatment effect between primary studies (heterogeneities). As different studies are not conducted by the same protocol, they will vary in the characteristics of the included population, in the concomitant interventions, in the diagnostic methods to access the outcomes, etc. (clinical heterogeneity). Studies may also vary in relation to their methodological characteristics. In the case of RCTs, we have included studies with higher or lower risk of bias, as studies that did not use Intention-to-treat analysis for example, or who have not had allocation concealment list. This type of variability is called methodological heterogeneity.

Statistical heterogeneity is the variability in effect estimation among studies included in the review. Both clinical heterogeneity and methodological heterogeneity are sources of statistical heterogeneity. However, a meta-analysis that has a significant clinical heterogeneity will not necessarily have a considerable statistical heterogeneity and, in these cases, the meta-analysis suggests a greater generalization capacity of the findings, increasing the external validity of the effect of the intervention. When heterogeneity is significant and can not be explained by any sensitivity analysis, meta-analysis is not recommended, and estimates of the effects of study interventions should only be presented individually. Some statistical tests can measure the degree of heterogeneity between studies. A large statistical test is the Higgins or I2 inconsistency test. It describes the percentage of variability in the estimate of effect that is attributed to heterogeneity rather than chance or chance.

Reference:

Higgins, J. P. T., & Green, S. (2003). Considerations and recommendations for figures in Cochrane reviews: graphs of statistical data. Cochrane Handbook for Systematic Reviews of Interventions. Version, 4(5).

Confounding of the statistical heterogeneity can be a significant flaw of meta-analysis.

From a methodology standpoint, I don't really see a downside if the meta-analysis is conducted correctly. I have read numerous meta-analyses that failed to perform any subgroup analysis. The danger in this is assuming the overall effect size reflects everything going on, when moderator variables can drastically change the results. The other "downside" might be that it will likely be time consuming.

Dear experts,

I hope this message finds you in good health and spirit.

I need a simple and easy protocol to perform meta-analysis. Can you suggest me a simple way to learn meta-analysis? Or can I have your protocol for meta-analysis?

I hope to receive good news from you.

All the best,

-Hassan

[email protected]

Dear Dr. Nagendra Nath Das

Many thanks for your kind attention. Your link was useful in which I found its attached paper amazing.

Hope all the best.

When selecting studies for mete-analysis if proper criteria is not used there may be issue of duplication of data when the results are published.