Deficiencies in systematic reviews and meta-analyses are not due to the methodology but rather to the people that apply the methodology. Of course, that could be said about any statistical procedure. I've reviewed manuscripts where researchers apply t-tests erroneously. If you think you see more errors in systematic reviews and meta-analyses than in other types of studies, you can blame the manuscript reviewers. Most are not trained on the methodology used in systematic reviews and meta-analyses. I've taught systematic review/meta-analysis procedures along with those for traditional statistics for over 13 years. Systematic review and meta-analysis procedures are not inherently more difficult or complex than other statistical procedures.

This is a great point. One of the biggest issues is that studies are not designed to ultimately be incorporated into systematic reviews, and therefore it becomes inherently difficult to compile a homogenous population of studies. Even with the highest quality studies, there will be inherent differences in populations, techniques, outcomes chosen to be reported, etc.

One of the most important things to do is provide metrics that allow for clarity. When there is significant study heterogeneity, researchers should not pool the data and claim statistical difference. Rather, forest plots depicting effect sizes with confidence intervals are more appropriate. Ranges, instead of pooled values, should be provided to avoid misinterpretation bias and inappropriate conclusions.

There are certainly still value in systematic reviews and meta-analyses if results are properly reported and methodological quality assessments are provided.

In my view the increasing amount of SRs is very much comparable with any product that we find on our market. Let's maybe take the example of clothes that we buy and wear every day. We need them, but the market is huge, since the demand is high. This leads to a high increase in supply and also a high loss of quality, since a lot of companies compete with each other and the buyer demands lower prices. Nevertheless, one might rather go for the high quality.

SRs on the other hand experienced increasing popularity, since they are crucial to make informed decisions for practitioners in medicine. SRs are the gold standard in the evidence pyramid. Of course, everyone tries to go with this trend. Problematic is now that there is no standardized quality for SRs. This leads to a confusing web of SRs that are often flawed. Thus, quality standards and transparency need to be enhanced. Some institutions like Cochrane are using strict quality standards, but on the other hand, these reviews often take more than a year. This might lead to that funding organs rather prefer to pay other institutions and also commercial companies that provide faster, but with lower quality. So, like with the clothes. We need them and we can get them everywhere, but we need to decide what’s more suitable for our own context in the long run. Low quality or high quality:

But there is no way to stop the' production' of SRs!

Deficiencies in systematic reviews and meta-analyses are not due to the methodology but rather to the people that apply the methodology. Of course, that could be said about any statistical procedure. I've reviewed manuscripts where researchers apply t-tests erroneously. If you think you see more errors in systematic reviews and meta-analyses than in other types of studies, you can blame the manuscript reviewers. Most are not trained on the methodology used in systematic reviews and meta-analyses. I've taught systematic review/meta-analysis procedures along with those for traditional statistics for over 13 years. Systematic review and meta-analysis procedures are not inherently more difficult or complex than other statistical procedures.

Seems inevitable wrt present scenario.

Personally I don't think it's a problem per se to keep writing systematic reviews and meta-analyses. In medicine at least, they provide invaluable information and guides both clinical practice and future research. I agree the quality of meta-analyses needs to be upheld, perhaps by more stringent enforcement of the PRISMA guideline, and I agree with Kyle N Kunze that part of that quality problem lies in the inconsistent reporting of outcomes and study designs of original studies. the EQUATOR network guidelines are probably a good way to sort this out, and more journals are requiring authors to follow these guidelines. In my field, there are also consensus documents outlining the variables to be reported for specific procedures, so I would eagerly anticipate more uniform reporting in the near future which should make our (reviewers') lives easier.

Another thing that is very important for me though is to consider the significance and potential impact of a meta-analysis (or systematic review). Is the subject important? Is it logical? What does it change? We see a lot of meta-analyses about specific genes and alleles, and many others on weird, niche topics of little (if any) clinical significance. Yes pretty much any study that gets written can be published somewhere eventually, but sometimes I feel that it just shouldn't be. In my opinion systematic reviews and meta-analyses should be reserved for important topics that has seen major breakthroughs, or that have seen a large body of evidence added to the literature without a systematic review having been done in a long while, e.g. 5 years. Then it has impact, and relevance. Otherwise a systematic review / meta-analysis is just largely pointless.

My two cents, there you go

In my point of view (Sport Science student), nowadays we've got over-information not just because of internet, also because the number of researchers has increased a lot. For example, in Spain, the number of Pdh students was a 10,9% higher from 2016 to 2017 (Spanish National Institute of Statistics data).

So a good proffesor or researcher should know through his experience which SR or MA is reliable. Also, Universities need to strenghten this critical analysis to their students in order to avoid this problem

I think SR and MA is instead further needed under the exponential increase in number of original research. Because we do not have so much time to go through the numerous amounts of original articles published. Paper publications seem to be out of favour under the era of internet, and with COVID-19, this is more advancing, and shifting many publication to pre-print without peer-review.

Article Early in the epidemic: impact of preprints on global discour...

And the tips lie on how do you choose SR and MA to support your research questions. Most SR and MA rely on those popular databases indexed original articles. Despite this, you can see that a MA usually trims down to less than 50 studies from few thousands to start with. I think the most important gate keeping step in keeping the quality depends on how these popular databases index each of their journals, thus limiting the amount of quality study to be "academically" available.

Reference to support as below;:

Severin, A., Low, N. Readers beware! Predatory journals are infiltrating citation databases. Int J Public Health 64, 1123–1124 (2019).

https://doi.org/10.1007/s00038-019-01284-3

Dear Colleague,

Your point is quite interesting. Undoubtedly systematic reviews and meta-analyses can contribute a high level of evidence. But, the authors should carefully follow methodological guidelines. In addition, meta-analysis should not be conducted if data synthesis is not indicated. Meta-analyses are often misleading. However, a carefully conducted systematic review without meta-analysis could give a more objective representation of current knowledge on a certain topic.

Systematic reviews with or without meta analyses are only as good as the literature they contain. Should low quality studies with a high risk of bias be included in this methodological process?

I note the points raised above, and it seems accepted that SR and MA are inevitable. However, would it may be prudent to have a registry for meta-analyses in particular and incorporate 'deciison making tools' at each step of the way when conducting a meta-analysis. Jeffrey Shi Kai Chan notes the PRISMA guidelines, and Cochrane style reviews have also been mentioned - their software is very user friendly, too. Scales that rate methodological quality pertinent to the field of interest are also a useful inclusion (e.g. PEDro or similar), and can be further supported in the statistical approach with tests such as heterogeneity, clearly defined boundaries for descriptions/magnitudes of effects, a clear presentation of confidence intervals, and acknowledgement of publication bias.

Hello all: This discussion started under the initial post and premise that the quality has deteriorated over the last two detects in a noticeable manner. I am curious as to how this conclusion was reached? Research, opinion, etc?

I think Jeffrey Martin makes a very good point - we should not assume that quality has deteriorated. Many of the issues raised now were prevalent before systematic reviews became common across many disciplines, including healthcare. The maxim 'garbage in, garbage out' has always applied to such analyses. I think that it is almost certainly true that with the increasing number of SRs the number of bad reviews has increased (even if their proportion hasn't). I observe, as a journal editor, that we see more and more reviews that prioritise process over content, and meta-analyses which seem to be largely undertaken 'because you can', rather than because there is a plausible possibility of drawing an inference about some meaningful population parameter. I don't have a good source to hand but I think there have been repeated studies showing that the reporting of reviews remains sub-optimal - which is, to some extent, a different issue as many of the 'brain dead' reviews (as I have come to refer to the process-driven activities I refer to above) are perfectly well reported.

I am not sure that the quality of systematic reviews (SRs) has deteriorated. They have likely had always been a summary of studies that have low methodological quality and a high risk of bias. The problem is the wide variability of tools that are used to assess the methodological quality of the studies contained within SRs and the overall poor methodological quality of the SRs as assessed with the AMSTAR 2 and other instruments. We have a paper that is presently in press that has looked at this. We need to have a discussion of when SR quality does not match the purported level of evidence.

This may be helpful for some folks:

Ekkekakis, P. (2015). Honey, I shrunk the pooled SMD! Guide to critical appraisal of systematic reviews and meta-analyses using the Cochrane review on exercise for depression as example. Mental health and physical activity, 8, 21-36.

All of the above comments are pertinent. One concern I have is that despite the review being poorly executed or questionable reporting, fellow reviewers often applaud the paper. Hence they are published (in reputable journals).

"Spin" is a well-documented phenomenon in the literature. In my opinion, it is the responsibility of the journal editors to ensure that the discussion is consistent with the data from the results.

The issue you have raised is of high importance in the scientific literature and some authors even call it "meta-analysis metastasis".

In my opinion, there are two ways to tackle it:

• Improving the quality of future systematic reviews and meta-analyses (SR&MA) by ameliorating the methodological background/education of researchers (even stressing on the fact that the combination of redundancy and poor quality research can be highly detrimental for science).
• Improving analytical skills of scientists and policymakers in such a way as to allow them to distinguish poor from high-quality SR&MA, thus making them capable of identifying reliable reviews among others.

With this regard, we have recently published an easy-to-apply guide to evaluate the quality of literature reviews. Feel free to use it:

Article Reading and interpreting reviews for health professionals: a...

Just curious as to how these recommendations are different or better than using the AMSTAR 2?

Personally, I find great knowledge in the systematic reviews that I read. I am usually exploring a new field of study and am ready to learn. I feel that I can sort out any bias of an author if there is any at all. I feel most authors are trying to teach about their areas of interest and also by doing so are able to bring into play some of what their intense research has found in their own labs and I appreciate this and often go and read their original papers on their findings.

I write reviews and my intentions are to summate what I have found that may have taken me months to learn and that if I can craft a paper such that in one or two evenings or one setting may set a young researcher, post-doc or fellow to condense their learning. I think in my heart of hearts we should keep writing reviews and systemic reviews to assist others in their quest of knowledge.

This is a good question Helmi Chaabene and I congratulate you on asking this and I thank you for posing this question and will enjoy reading the many answers and thoughts that this question provokes.

Melvin R Hayden, MD

University of Missouri School of Medicine

Columbia, Missouri

Dear colleagues,

I really enjoyed reading your great answers. I can see that you all agreed about the fact that SRs & MA are inevitable. However, it is more about the way we are approaching SR & MA that needs much of our attention. In other words, a clear and concise question should be asked and a sound/rigorous methodological approach needs to be adopted. We, as researchers, reviewers, and may be editors have a great responsibility for making sure about the quality of submitted SRs & MA.

Most importantly the SRs with and without MA should follow the PRISMA guidelines to ensure transparent reporting and be prospectively registered through PROSPERO to ensure that the authors followed their prospective intent. Did the authors perform the review to answer a question or support an answer? If this question cannot be answered the validity of the SR cannot be determined.

in my opinion, well- qualified meta analyses should be requested by special institutes or some journals in order to choose experts for writing them, nowadays we can see worthless reviews which don't follow Prisma. in any case, the volume of recent reviews is more than original articles which are more important to carry out.....

Misconducted systematic reviews provides misleading findings like as misconducted clinical trials provides misleading inferences. The problem is not the type os study but who conduct them.

This paper discusses the issue. Article Should low-quality evidence dominate high-level evidence? A ...

The major assumption here is that the individuals conducting the SR with or without a MA (SR/MA) know and understand that there is a registry, reporting guidelines, qualitative standards, and gold standards for conducting this type of research. That said, I would suggest that it is the responsibility of the journal editors to ensure that this type of research is registered, registered prospectively and that the registry matches the published product. This is not happening! Additionally, reviewers need to ensure that the authors followed reporting and qualitative guidelines methodologically. Journal editors are always chasing a journal impact factor. It is believed (whether accurate or not) that higher levels of evidence such as SR/MA and RCTs are more likely to be cited and are therefore more likely to get published. We have plenty of quantity. We need more easily consumable quality for the working clinician that does not have time to take a deep dive into a study's validity. Just my opinion. Use if helpful.

My answer is NO, we should keep conducting high-quality systematic reviews/meta-analyses. Systematic reviews/meta-analyses are very useful for informed decision process and should be updated regularly, considering new high-quality studies and summarizing the current quality of evidence for a specific question (e.g., efficacy of interventions, prognosis, risks, among others). However, I believe that researchers, clinicians, policemakers and patients should be careful when conducting or reading systematic reviews as too many of them have serious flaws. To conduct or read them, people need to understand evidence-based practice and how to read a paper effectively (critical appraisal of the study).

Nowadays the usefulness of review is much more lower than decades ago. In age of Internet everybody can perform any "review" just looking in any search engine.