Very interesting discussion! I think another point is that SR & MAs often pool studies that should not be pooled, which is not necessarily caught by quality assessment scales and measures of heterogeneity (see for example the discussions about which measure of heterogeneity to use or not to use: Article Basics of meta-analysis: I(2) is not an absolute measure of ...

Article Can Meta-analyses be trusted

Thank you for your nice comment Fahimeh Haghighatdoost . You are exactly right. But I think the journals should seek for true evidence and also put more effort to control the quality of their publications. Particularly systematic reviews and/or meta-analyses, by asking for data and assessing the search strategies and the statistical analyses. These type of studies significantly affect the practice and misleading ones might put burden on societies.

Hello Amin

It is difficult to disagree with your assessment of many current SRs. Yet, if you went back say, 5 or 10 years and looked at the SRs published at the time, they are hardly any better. I would say that there has been a vast imrpovement since the coming of the MOOSE -> QUORUM -> PRISMA reporting formats. I can't imagine any quality journals within health sciences today will accept an SR that has not adopted to PRISMaA. Moreover, some even insist on prepub registration of SRs in PROSPERO before considering reviewing a submission. Now, a meta-analysis is never better than the primary studies and the frequent inadequate reporting is really frustrating when attempting to interpret the reported primary data. I would venture that some of the explanation of "poor SRs" today is also due to inadequate reporting of the primary data in the first place. Moreover, there is surprisingly little focus on the risk of type I errors in meta-analyses, and far more authors should adopt trial sequential analysis (TSA) of cumulative meta-analyses to account for random errors and study heterogenity. Finally , the term "true evidence" should be shunned as it is only an abstraction in the mindset of believers in scientism. Progressive scientists are in their soul true believers of rational scepticism.

Hello Asbjørn

Thank you for your feedback. I agred with you that the systematic reviews have been improved since 5-10 years ago and the reporting guidelines like PRISMA, MOOSE and etc. have tried to improve the systematic reviews and meta-analyses. However thsee reporting formats does not guarantee the quality of a work. A low quality systematic review and meta-analysis can get published in a high impact journal even by following PRISMA and being registered in PROSPERO. A systematic review might ahave a low quality search strategy, do not completely define the statistical methods but become registered in PROSPERO. I Think that the review process is poor and also the guidelines should be improved. For instance, asking for the data extracted from included studies and checking if the data are properly extracted by a reviewer and running the analyses could guarantee the statistical analysis part. The data can also be provided for the public. Furthermore, I think providing the exact search strategies which can be applied in databases should be mandatory. Therefore, the search strategy can be critically reviewed. The reporting guidelines do not ask for these items at this time. It would be interesting to have guidelines for reviewing the systematic reviews and meta-analyses.

Furthermore, I beleibve that journals should welcome the manuscripts that reject or improve the results of their own publication.

I think as researchers we all need to be smart enough to judge the quality of published articles before we draw inference from the findings of that particular paper. Despite our best efforts the possibility of withdrawing or eliminating such reviews is very slim, however, it's up to us to utilized our research skills the best way we can to ensure that whatever findings we infer from is factual and precise and I believe that we have quite a few tool to assess the quality of systematic review and meta analysis.

Thanks Theosophist,

I agree with your comments. Of course we have to be smart enough as a researcher; but how about practitioners? They hardly understand the methodology. They know that systematic reviews and meta-analyses are at the top in hierarchy of evidence and they usually rely on the results found by systematic reviews and meta-analyses published in high impact journals.

Amin Salehi-Abargouei we do have evidence review guide for clinicians as well in which the criteria for good quality review was summarized in one table and they can used that to appraised the review before inferring form it. I will send it to you maybe it will help.

Thank you so much Theophilus Lakiang . May I ask you to attach the file here in you reply? and also attach any paper which has assessed the tool?

Amin Salehi-Abargouei I will do that, quite busy at the moment and have to search for it cuz am not very sure of the source but I have included it once in my presentation.

Hello again Amin

I believe that we agree on most of the arguments that you bring up.

First – to assist you with your search for guidelines, you will find 34 suggestions for evaluating health research SRs& MAs on the Equator network website, there are 34 http://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_study_design=systematic-reviews-and-meta-analyses

The number of proposed guidelines should be a precautionary indication that the magic bullet has not been identified yet.

Secondly, in your invitation for discussion, you use the label "quality" quite freely, but you have to realize that this term is ambiguous. You and the readers of your comments are not necessarily in synch in the translation of fictions like "highest quality of evidence" , "quality of the systematic reviews", "low-quality systematic reviews and meta-analyses", "high-quality search strategies and data analyses", "quality assessment", "high-quality systematic reviews and meta-analyses" and "high-quality journals". Hence, since these terms are ambiguous, an invitation for discussion may be lost in translation and end up becoming another exercise in tautology.

Third, in my opinion best SRs & MAs present published and unpublished evidence, explain how the evidence was located, identify probable biases associated with the evidence, and aggregate the evidence if meaningful and not too heterogeneous. As an inexperienced academician I joined the ranks of upset individuals when two experts smarter than me, i.e., Peter Gøtzsche and John Ioannidis argue that it is not always sensible to include subject experts as authors of systematic reviews and meta-analyses (https://doi.org/10.1136/bmj.e7031). Today, I understand better because judging the Quality of evidence is on a higher dimension and even one notch higher again is that the interpretation of presented evidence must be judged in context of content expertise.

Re. high-quality search strategies I am amazed that individuals do not just adopt the Cochrane collaboration for any topic in health sciences. These search strategies have high sensitivity and specificity and it is just a matter of removing the "RCT"-filter.

I am intrigued by your statement that "high-quality systematic reviews and meta-analyses which reject the results of the previous misleading systematic reviews or improve the previous evidence, hardly get published." I would appreciate if have can provide a citation for this statement.

Your suggestion that we need high-quality journals specifically designed to publish systematic reviews without considering their readership, their results and the possibility of getting citations is excellent. The idea was actually the original core of both the Cochrane Collaboration for health sciences and the Campbell collaboration for social sciences. In the early 90ies when the Cochrane movement began, there was some anarchy in that many SR authors instead of just ending with a statement of the state of the evidence provided guidance on how the reader should interpret the evidence. Then the collaboration decided in 2002 in Stavanger that the SRs should contain minimal clinical guidance and limited to only presenting evidence. Alas, today we are back to the "trusted evidence" from the Cochrane collaboration which accounts for many tens of millions GBP annually from funding and support. Take a guess whether the funders and sponsors pay for products that contain descriptions of the (often inadequate) level of evidence, or a product that contain clinical guidance when deciding on NGO and national health policy decision making.

Hello Asbjørn

Thank you again for sharing with us your information in this discussion.

Thank you for introducing equator and the guidelines provided for reporting different systematic reviews and overview of systematic reviews. As I mentioned before, these tools improve the reporting not quality of the work. I think I have to explain about what I do mean by saying low quality systematic reviews. The systematic reviews that do not try to find, select, appraise and synthesize the original studies correctly. These types of studies might report their investigation based on the available reporting guidelines but we you look at their approach is weak. Therefore we can see publications which have used PRISMA, MOOSE, etc. to report the study but they have led to misleading results.

Let’s provide you two examples. If you see the following papers you can see that the SRM found that L-carnitine significantly reduces body weight and the other study which is a comment on the systematic review and meta-analysis rejected the results of the first one.

Systematic review and meta-analysis: https://www.ncbi.nlm.nih.gov/pubmed/27335245

Comment:

https://www.ncbi.nlm.nih.gov/pubmed/28008707

Fortunately, the comment was published in this case. The first paper is reported based on PRISMA guideline. Interestingly, we are conducting SRM on the same subject and found that both these two studies have missed more than 10 studies.. We are now finalizing its manuscript.

Example two: We found two studies which examined the effect of vitamin D supplementation on serum leptin and adipnectin levels:

https://www.ncbi.nlm.nih.gov/pubmed/27966571

https://www.ncbi.nlm.nih.gov/pubmed/27038530

In comparison to the above mentioned meta-analyses conducted by Dinca et al and Hajimohammadi et al, we found 13 and 16 more studies respectively. We also concluded that the previously published systematic reviews had limitations in their search strategy; and based on the current evidence, vitamin D supplementation does not have a significant effect on adiponectin and leptin levels. Our study is under consideration at this time; however, we received several rejections. The above-mentioned studies were also reported based on PRISMA guideline. I name these kinds of mistakes as low quality.

I agree with all of your other comments particularly the last paragraph.

Thanks again

I really like Dr Asbjorn's comments. Yes, it is interesting that Cochrane Collaboration database is not used as much.

The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses: https://www.milbank.org/quarterly/articles/mass-production-redundant-misleading-conflicted-systematic-reviews-meta-analyses/

What a wonderfull paper! Thank you for making us aware of the contents. I speculate that since Ionnidis attracted so much hostile remarks when he authored the paper titled "why most published research findings are false" that he didn't follow up with the title : "why most published sr-ma's are both false and fake? Happy weekend!

Thanks Claire, excelent review?

MILBANK QUARTERLY's Article Influence Score (AIS) is 3.2, which was the best AIS among any other journals. Based on the score, we can tell that the paper is very influential. And I do not highly think of scholars who mostly publish such a review paper or meta-analysis paper. An excellent scholar have his/her own specialized original and creative article.

Thank you Claire, for this nice paper. Ionnidis has thoroughly discussed the point. I think it will be nice to discuss about how we can improve the situation!

Dear Colleagues,

Excellent discussion!

Congratulations!

Good discussion following the timeous critique on sr-ma

I agree

Thank you for this very good discussion

Excellent discussion

Excellent dear Dr.

I wish you the best.

Dear Sarmad Sobhi Al Qassar a plethora of guidance is available today for reporting, methodological quality and assessment of risk of bias of systematic reviews with meta-analyses (indeed, not so new). Please check the AMSTAR-1 and AMSTAR-2, PRISMA and extensions (to guide appropriate REPORTING and not to assess methodological quality neither the proper reporting) and ROBIS, for example (among a great variety then). Cheers, LH.

Dear Colleagues,

I especially agree with the fact that at least 4 referees should be required to evaluate systematic reviews and meta-analyzes: one for critical assessment of the search strategies, one to assess the statistical methods, one to assess the methodology and one to assess the scientific content of the submitted papers.

Thank you for opening this debate!

This is a very good point. Please provide or better publish a score sheets (e.g., considered Databases, search strategy (including proper keywords), search report, analyses adjusted for small samples, heterogeneity, proper overview of indlucded studies, funnal plots, double reporting, pooling of outcomes etc.) that allows editor to "quick-screen" submitted Reviews or Meta-analyses

Dear all- thank you for this discussion- I guess it is really necessary to double check the quality criteria of the review alteady within the reviewer state. I got some articles as a reviewer that pointed put that they used the PRISMA guidelines for but when I controlled them, the manuscript did not include relevant aspects. We can improve the quality of future publications if we take our time for contructive feedback to the authirs.

time for constructive amd good geedback time

sry- the app does not work well- could not correct any spelling im my answer...

Not surprising, and somehow courageous, DoveMedicalPress, an open-access publisher, as per April 2019, does not accept SRs and MAs anymore.

Very interesting discussion! I think another point is that SR & MAs often pool studies that should not be pooled, which is not necessarily caught by quality assessment scales and measures of heterogeneity (see for example the discussions about which measure of heterogeneity to use or not to use: Article Basics of meta-analysis: I(2) is not an absolute measure of ...

Article Can Meta-analyses be trusted

Dear colleagues,

Because of my concerns about the review articles, particularly systematic reviews and meta-analyses, I recently tried to start a journal [critical comments in bio-medicine (CCB)] for publishing review articles. The journal is open access and also does not charge any publication or processing fee. I have tried to use Open journal systems to handle the journals website to reduce the costs.

I will be happy to have your comments about the journal and have you by my side to handle the journal. I have tried to include some of interested researchers in the Editorial board. I also have sent several invitations for researchers that I thought that they might be interested. We are open for adding interested researchers particularly those experienced in systematic reviews and meta-analyses and have concern about this, in the editorial board. I think this might help in improving the quality of papers. The journal's website is https://www.ccbjournal.info

As I mentioned in the about the journal section, our team believes that professional journals like CCB might better evaluate such articles by gathering the experts interested in the field which makes the published evidence more reliable. The journal is dedicated to publishing review articles, only considering their quality without taking the manuscripts’ readership, their results and the possibility of getting citations. Therefore, we will be happy to have your manuscript published in CCB. We will not hurry for publishing our first issue and will wait for manuscripts with minimum standards. We will also try our best to increase our standards.

Please feel free to contact me using researchgate or my email (abargouei[at]ssu.ac.ir) if you think that you can help me in this.

Amin Salehi-Abargouei I've been working hard with predatory journals and editorial process with David (Moher) and our group. Giving your speech and taking a look at your website (in each section), I would be happy to give you some inputs to improve your editorial process and other aspects of the journal. Could you write me at [email protected] and then we can try to arrange a way to talk more about? I'm wishing all the best to your initiative! Cheers, LH.

There is at least one serious journal devoted to systematic reviews in the field of environmental sciences: https://environmentalevidencejournal.biomedcentral.com/

Also, have a look at these two papers on meta-analysis:

https://www.nature.com/articles/nature25753

Article Meta-evaluation of meta-analysis: Ten appraisal questions fo...

Cheers,

alex

Our perspective on the shortcomings of meta-analyses, in particular with respect to supplementation studies @ https://www.germanjournalsportsmedicine.com/archiv/archive-2020/issue-2/vitamin-d-and-athletic-performance-perspectives-and-pitfalls/

"...several aspects should provide caution against oversimplified inference when interpreting the conclusions derived from meta-analyses on supplementation studies"...

As far as systematic reviews are concerned, in the last few years (and I have no doubt that with positive intentions) a large amount of "methodological bureaucracy" has been generated which may give the appearance of science, but at the same time hide some not very relevant findings. In other words, I have seen some perfect revisions from the formal point of view but empty of content and, above all, empty of relevance.

I totally agree. Even when findings are not relevant or point out the opposite direction, they should be included. Negative findings are, as well, important to guide science in the good direction and generate fruitful projects instead of picking the same stone once and again because nobody published that there was nothing to search there or other variables should be considered.

I share many concerns that have been expressed above. At the moment, I am reading a review in which the criteria that have been set for including studies in the review are highly questionable, as are the evaluation criteria for the studies that were retained.

In essence, the reviewers are good at making reasonably simple "in-out / yes- no" decisions, but they do not demonstrate sophistication in terms of scientific methodology or statistical validity.

Recently, I have read two other systematic reviews in which the reviewers exhibited the same deficiencies.

These are important issues, and I'm gratified to see others in this thread airing similar issues about reviews.

Absolutely agree with respected Robert Trevethan in very rightly pointing out " they do not demonstrate sophistication in terms of scientific methodology or statistical validity" as some of the researchers simply think it is easy to write a "Systematic Review" on a topic without even understanding the purpose (and process) of doing so.

Amin Salehi-Abargouei has picked up an important topic of discussion because Systematic Reviews and Meta-analyses provide the foundation for the development of knowledge in a given field.

Low quality reviews of individual healthcare mobile apps based on flimsy criteria or mere personal opinion are being swept into "systematic" reviews. We proposed an ecological solution where careful reviews of apps are collected into an online library where they may be compared, reasonably aggregated and scientifically analyzed. Our approach is at doi:10.1007/s41347-016-0005-z and doi:10.1007/s41347-017-0011-9 .

Hi-

It is really critical that the quality improves- I think coming up with a system which is more consistent with GRADE or JBI which downgrades poor quality RCTs.

A related issue is "propping up" clinical practice guidelines (CPGs) as best evidence. From my perspective, CPGs add another layer of professional opinion between actual high quality research studies and the sheepish practitioner who may not have the skill set - they just follow the herd. I am concerned about CPGs being purported as high level evidence when in fact they represent an expert panel's interpretation of evidence. This may sound like hair splitting but its an important consideration in the debate regarding expectations of health professionals using professional thinking and professional skills sets versus "technicians" who provide services based on an expertise hierarchy. This trend is disturbing to me.

The inherent defect in a Clinical Practice Guideline (CPG) system does indeed resemble the basic weakness in the fields of systematic reviews and meta-analyses. Generating CPGs is a political process cloaked in science. Implementation of what seems a well-intentioned beneficial scheme quickly creates major flaws, generates unintended adverse consequences and opens opportunities for predation. Efforts to mitigate the problem with more caucusing, bureaucratic activity and punishment worsen their impact. Yes, this is disturbing.

This is true. The main reason is that the reviewers that are reviewing are not always the best.