Dear Colleagues,
Systematic reviews and meta-analyses are supposed to provide the highest quality of evidence. The number of systematic reviews and meta-analyses are increasing in different fields. Although several reporting criteria are provided to increase the quality of the systematic reviews we see that the number of low-quality systematic reviews and meta-analyses are also increasing. The majority of these studies lack high-quality search strategies and data analyses. The quality assessment is not properly done. I can see the low-quality systematic reviews and meta-analyses get published just because they have provided a significant (and also misleading) evidence and the editors always like the significant and also surprising evidence. Particularly this evidence are so welcomed because they will bring much more citations. The surprising part is that the high-quality systematic reviews and meta-analyses which reject the results of the previous misleading systematic reviews or improve the previous evidence, hardly get published. I think at this stage we need high-quality journals specifically designed to publish systematic reviews without considering their readership, their results and the possibility of getting citations. These journals need to put more time on assessing the provided evidence and have at least four referees: one for critical assessment of the search strategies, one to assess the statistical methods, one to assess the methodology and one to assess the scientific content of the submitted papers. Furthermore, a specific checklist is needed to be designed for reviewing such papers. I wanted to know your opinions and experiences in this regard.
Very interesting discussion! I think another point is that SR & MAs often pool studies that should not be pooled, which is not necessarily caught by quality assessment scales and measures of heterogeneity (see for example the discussions about which measure of heterogeneity to use or not to use: Article Basics of meta-analysis: I(2) is not an absolute measure of ...
Article Can Meta-analyses be trusted
). The intended use of SR and MAs is to "summarize the findings of several clinical trials that evaluate the effectiveness or efficacy of a similar treatment approach on similar group of patients", and if it is done in such manner it can provide valuable aggregated information. However, I have the impression that the term "similar" is often used very loosely, leading to the problem that many SR and MAs compare apples and oranges, resulting in misleading conclusions or inconclusive findings. An issue that adds to this problem is that in some fields replication studies are not encouraged (i.e., are much more likely not to be published because they are not novel), even though replicated or very similar studies are the basis of reliable SRs and MAs.
Thank you for your nice comment Fahimeh Haghighatdoost . You are exactly right. But I think the journals should seek for true evidence and also put more effort to control the quality of their publications. Particularly systematic reviews and/or meta-analyses, by asking for data and assessing the search strategies and the statistical analyses. These type of studies significantly affect the practice and misleading ones might put burden on societies.
Hello Amin
It is difficult to disagree with your assessment of many current SRs. Yet, if you went back say, 5 or 10 years and looked at the SRs published at the time, they are hardly any better. I would say that there has been a vast imrpovement since the coming of the MOOSE -> QUORUM -> PRISMA reporting formats. I can't imagine any quality journals within health sciences today will accept an SR that has not adopted to PRISMaA. Moreover, some even insist on prepub registration of SRs in PROSPERO before considering reviewing a submission. Now, a meta-analysis is never better than the primary studies and the frequent inadequate reporting is really frustrating when attempting to interpret the reported primary data. I would venture that some of the explanation of "poor SRs" today is also due to inadequate reporting of the primary data in the first place. Moreover, there is surprisingly little focus on the risk of type I errors in meta-analyses, and far more authors should adopt trial sequential analysis (TSA) of cumulative meta-analyses to account for random errors and study heterogenity. Finally , the term "true evidence" should be shunned as it is only an abstraction in the mindset of believers in scientism. Progressive scientists are in their soul true believers of rational scepticism.
Hello Asbjørn
Thank you for your feedback. I agred with you that the systematic reviews have been improved since 5-10 years ago and the reporting guidelines like PRISMA, MOOSE and etc. have tried to improve the systematic reviews and meta-analyses. However thsee reporting formats does not guarantee the quality of a work. A low quality systematic review and meta-analysis can get published in a high impact journal even by following PRISMA and being registered in PROSPERO. A systematic review might ahave a low quality search strategy, do not completely define the statistical methods but become registered in PROSPERO. I Think that the review process is poor and also the guidelines should be improved. For instance, asking for the data extracted from included studies and checking if the data are properly extracted by a reviewer and running the analyses could guarantee the statistical analysis part. The data can also be provided for the public. Furthermore, I think providing the exact search strategies which can be applied in databases should be mandatory. Therefore, the search strategy can be critically reviewed. The reporting guidelines do not ask for these items at this time. It would be interesting to have guidelines for reviewing the systematic reviews and meta-analyses.
Furthermore, I beleibve that journals should welcome the manuscripts that reject or improve the results of their own publication.
I think as researchers we all need to be smart enough to judge the quality of published articles before we draw inference from the findings of that particular paper. Despite our best efforts the possibility of withdrawing or eliminating such reviews is very slim, however, it's up to us to utilized our research skills the best way we can to ensure that whatever findings we infer from is factual and precise and I believe that we have quite a few tool to assess the quality of systematic review and meta analysis.
Thanks Theosophist,
I agree with your comments. Of course we have to be smart enough as a researcher; but how about practitioners? They hardly understand the methodology. They know that systematic reviews and meta-analyses are at the top in hierarchy of evidence and they usually rely on the results found by systematic reviews and meta-analyses published in high impact journals.
Amin Salehi-Abargouei we do have evidence review guide for clinicians as well in which the criteria for good quality review was summarized in one table and they can used that to appraised the review before inferring form it. I will send it to you maybe it will help.
Thank you so much Theophilus Lakiang . May I ask you to attach the file here in you reply? and also attach any paper which has assessed the tool?
Amin Salehi-Abargouei I will do that, quite busy at the moment and have to search for it cuz am not very sure of the source but I have included it once in my presentation.
Hello again Amin
I believe that we agree on most of the arguments that you bring up.
First – to assist you with your search for guidelines, you will find 34 suggestions for evaluating health research SRs& MAs on the Equator network website, there are 34 http://www.equator-network.org/?post_type=eq_guidelines&eq_guidelines_study_design=systematic-reviews-and-meta-analyses
The number of proposed guidelines should be a precautionary indication that the magic bullet has not been identified yet.
Secondly, in your invitation for discussion, you use the label "quality" quite freely, but you have to realize that this term is ambiguous. You and the readers of your comments are not necessarily in synch in the translation of fictions like "highest quality of evidence" , "quality of the systematic reviews", "low-quality systematic reviews and meta-analyses", "high-quality search strategies and data analyses", "quality assessment", "high-quality systematic reviews and meta-analyses" and "high-quality journals". Hence, since these terms are ambiguous, an invitation for discussion may be lost in translation and end up becoming another exercise in tautology.
Third, in my opinion best SRs & MAs present published and unpublished evidence, explain how the evidence was located, identify probable biases associated with the evidence, and aggregate the evidence if meaningful and not too heterogeneous. As an inexperienced academician I joined the ranks of upset individuals when two experts smarter than me, i.e., Peter Gøtzsche and John Ioannidis argue that it is not always sensible to include subject experts as authors of systematic reviews and meta-analyses (https://doi.org/10.1136/bmj.e7031). Today, I understand better because judging the Quality of evidence is on a higher dimension and even one notch higher again is that the interpretation of presented evidence must be judged in context of content expertise.
Re. high-quality search strategies I am amazed that individuals do not just adopt the Cochrane collaboration for any topic in health sciences. These search strategies have high sensitivity and specificity and it is just a matter of removing the "RCT"-filter.
I am intrigued by your statement that "high-quality systematic reviews and meta-analyses which reject the results of the previous misleading systematic reviews or improve the previous evidence, hardly get published." I would appreciate if have can provide a citation for this statement.
Your suggestion that we need high-quality journals specifically designed to publish systematic reviews without considering their readership, their results and the possibility of getting citations is excellent. The idea was actually the original core of both the Cochrane Collaboration for health sciences and the Campbell collaboration for social sciences. In the early 90ies when the Cochrane movement began, there was some anarchy in that many SR authors instead of just ending with a statement of the state of the evidence provided guidance on how the reader should interpret the evidence. Then the collaboration decided in 2002 in Stavanger that the SRs should contain minimal clinical guidance and limited to only presenting evidence. Alas, today we are back to the "trusted evidence" from the Cochrane collaboration which accounts for many tens of millions GBP annually from funding and support. Take a guess whether the funders and sponsors pay for products that contain descriptions of the (often inadequate) level of evidence, or a product that contain clinical guidance when deciding on NGO and national health policy decision making.
Hello Asbjørn
Thank you again for sharing with us your information in this discussion.
Thank you for introducing equator and the guidelines provided for reporting different systematic reviews and overview of systematic reviews. As I mentioned before, these tools improve the reporting not quality of the work. I think I have to explain about what I do mean by saying low quality systematic reviews. The systematic reviews that do not try to find, select, appraise and synthesize the original studies correctly. These types of studies might report their investigation based on the available reporting guidelines but we you look at their approach is weak. Therefore we can see publications which have used PRISMA, MOOSE, etc. to report the study but they have led to misleading results.
Let’s provide you two examples. If you see the following papers you can see that the SRM found that L-carnitine significantly reduces body weight and the other study which is a comment on the systematic review and meta-analysis rejected the results of the first one.
Systematic review and meta-analysis: https://www.ncbi.nlm.nih.gov/pubmed/27335245
Comment:
https://www.ncbi.nlm.nih.gov/pubmed/28008707
Fortunately, the comment was published in this case. The first paper is reported based on PRISMA guideline. Interestingly, we are conducting SRM on the same subject and found that both these two studies have missed more than 10 studies.. We are now finalizing its manuscript.
Example two: We found two studies which examined the effect of vitamin D supplementation on serum leptin and adipnectin levels:
https://www.ncbi.nlm.nih.gov/pubmed/27966571
https://www.ncbi.nlm.nih.gov/pubmed/27038530
In comparison to the above mentioned meta-analyses conducted by Dinca et al and Hajimohammadi et al, we found 13 and 16 more studies respectively. We also concluded that the previously published systematic reviews had limitations in their search strategy; and based on the current evidence, vitamin D supplementation does not have a significant effect on adiponectin and leptin levels. Our study is under consideration at this time; however, we received several rejections. The above-mentioned studies were also reported based on PRISMA guideline. I name these kinds of mistakes as low quality.
I agree with all of your other comments particularly the last paragraph.
Thanks again
I really like Dr Asbjorn's comments. Yes, it is interesting that Cochrane Collaboration database is not used as much.
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta-analyses: https://www.milbank.org/quarterly/articles/mass-production-redundant-misleading-conflicted-systematic-reviews-meta-analyses/
What a wonderfull paper! Thank you for making us aware of the contents. I speculate that since Ionnidis attracted so much hostile remarks when he authored the paper titled "why most published research findings are false" that he didn't follow up with the title : "why most published sr-ma's are both false and fake? Happy weekend!
MILBANK QUARTERLY's Article Influence Score (AIS) is 3.2, which was the best AIS among any other journals. Based on the score, we can tell that the paper is very influential. And I do not highly think of scholars who mostly publish such a review paper or meta-analysis paper. An excellent scholar have his/her own specialized original and creative article.
Thank you Claire, for this nice paper. Ionnidis has thoroughly discussed the point. I think it will be nice to discuss about how we can improve the situation!
Dear Sarmad Al Qassar a plethora of guidance is available today for reporting, methodological quality and assessment of risk of bias of systematic reviews with meta-analyses (indeed, not so new). Please check the AMSTAR-1 and AMSTAR-2, PRISMA and extensions (to guide appropriate REPORTING and not to assess methodological quality neither the proper reporting) and ROBIS, for example (among a great variety then). Cheers, LH.
Dear Colleagues,
I especially agree with the fact that at least 4 referees should be required to evaluate systematic reviews and meta-analyzes: one for critical assessment of the search strategies, one to assess the statistical methods, one to assess the methodology and one to assess the scientific content of the submitted papers.
Thank you for opening this debate!
This is a very good point. Please provide or better publish a score sheets (e.g., considered Databases, search strategy (including proper keywords), search report, analyses adjusted for small samples, heterogeneity, proper overview of indlucded studies, funnal plots, double reporting, pooling of outcomes etc.) that allows editor to "quick-screen" submitted Reviews or Meta-analyses
Dear all- thank you for this discussion- I guess it is really necessary to double check the quality criteria of the review alteady within the reviewer state. I got some articles as a reviewer that pointed put that they used the PRISMA guidelines for but when I controlled them, the manuscript did not include relevant aspects. We can improve the quality of future publications if we take our time for contructive feedback to the authirs.
time for constructive amd good geedback time
sry- the app does not work well- could not correct any spelling im my answer...
Not surprising, and somehow courageous, DoveMedicalPress, an open-access publisher, as per April 2019, does not accept SRs and MAs anymore.
Very interesting discussion! I think another point is that SR & MAs often pool studies that should not be pooled, which is not necessarily caught by quality assessment scales and measures of heterogeneity (see for example the discussions about which measure of heterogeneity to use or not to use: Article Basics of meta-analysis: I(2) is not an absolute measure of ...
Article Can Meta-analyses be trusted
). The intended use of SR and MAs is to "summarize the findings of several clinical trials that evaluate the effectiveness or efficacy of a similar treatment approach on similar group of patients", and if it is done in such manner it can provide valuable aggregated information. However, I have the impression that the term "similar" is often used very loosely, leading to the problem that many SR and MAs compare apples and oranges, resulting in misleading conclusions or inconclusive findings. An issue that adds to this problem is that in some fields replication studies are not encouraged (i.e., are much more likely not to be published because they are not novel), even though replicated or very similar studies are the basis of reliable SRs and MAs.
Dear colleagues,
Because of my concerns about the review articles, particularly systematic reviews and meta-analyses, I recently tried to start a journal [critical comments in bio-medicine (CCB)] for publishing review articles. The journal is open access and also does not charge any publication or processing fee. I have tried to use Open journal systems to handle the journals website to reduce the costs.
I will be happy to have your comments about the journal and have you by my side to handle the journal. I have tried to include some of interested researchers in the Editorial board. I also have sent several invitations for researchers that I thought that they might be interested. We are open for adding interested researchers particularly those experienced in systematic reviews and meta-analyses and have concern about this, in the editorial board. I think this might help in improving the quality of papers. The journal's website is https://www.ccbjournal.info
As I mentioned in the about the journal section, our team believes that professional journals like CCB might better evaluate such articles by gathering the experts interested in the field which makes the published evidence more reliable. The journal is dedicated to publishing review articles, only considering their quality without taking the manuscripts’ readership, their results and the possibility of getting citations. Therefore, we will be happy to have your manuscript published in CCB. We will not hurry for publishing our first issue and will wait for manuscripts with minimum standards. We will also try our best to increase our standards.
Please feel free to contact me using researchgate or my email (abargouei[at]ssu.ac.ir) if you think that you can help me in this.
Amin Salehi-Abargouei I've been working hard with predatory journals and editorial process with David (Moher) and our group. Giving your speech and taking a look at your website (in each section), I would be happy to give you some inputs to improve your editorial process and other aspects of the journal. Could you write me at [email protected] and then we can try to arrange a way to talk more about? I'm wishing all the best to your initiative! Cheers, LH.
- Badges
- Science topic
- Similar topics
- Academic Journals