I want to assess the literature on what the evidence says about a specific relationship. Given my emphasis on "empirical generalization" rather than a full detailed set of empirical evidence, is it ok to do a systematic review of only the most highly cited articles (using some benchmark for the citation cutoff)? Is there any research that has taken this route?
Hi Srinavas
None that I know of, although my dealing with systematic reviews has been limited. However, you need to include some filtering to exclude "vanity" citations from cartels, citations for examples of what not to do, and the examples of fraud, stupidity, or simply erroneous methods or conclusions that litter every arm of literature. In some fields the "wrong-uns" outnumber the good ones.
Good luck, Ian
Hello Srinavas:
Let Cochrane Collaboration be you guide. To me Cochrane is the gold standard for systematic reviews.
Go here: http://www.cochrane.org/MR000026/METHOD_examining-reference-lists-to-find-relevant-studies-for-systematic-reviews
Good luck
Chuke
Systematic reviews require a certain formality in the process of selecting the articles for review. The selection process is usually thought of in terms of key word searches and discard rules. The formalism of the selection process increases the replicability of the study and clarifies the scope of the study ( e.g. whether commercially significant source material like patents were considered in scope for the study). These mechanisms are intended to build the credibility and value of the study. In principle, you could use selection criteria based from citation analysis, but this still results in a line being drawn between the papers included in the study and those discarded or excluded from the study and that line should be justified in some way in the research methods section of your study.
I think that in addition to the above-mentioned cochrane, you could use search engines like pubmed, scopus, thomson reuters, scielo, that combine articles of good technical quality
Thanks, Ian, Chuke, Steven and Luis. Each of your responses has given me a number of things on which to build robustness for my study.
A systematic review sets out to give an unbiased view of the outcome(reducing the risk of bias as much as possible)
two likely sources of bias you might introduce with the suggested approach are the fact that high impact journals sometimes favour larger effect sizes
http://www.thelancet.com/journals/lancet/article/PII0140-6736(91)90201-Y/abstract
another reason for going for citations based cut-offs is that citations favour often the predominant line of thought which might be more positive than the general result http://www.jclinepi.com/article/S0895-4356(16)30750-8/fulltext
If time, resources or other reasons prevent you from doing an exhaustive search across multiple databases choosing one database might help you more see e.g. https://www.ncbi.nlm.nih.gov/pubmed/27670136 for guidance
hope this helps
http://www.sciencedirect.com/science/article/pii/S1025984815000228
Today marks the release of the 2017 Highly Cited Researchers list. Now in it’s 4th year, the Highly Cited Researchers list represents some of the world’s most influential scientific minds. These researchers, spanning 21 fields of the sciences and social sciences, rank in the top 1% by citations for field and publication year in Web of Science. They are your colleagues, teachers, and mentors– and they leading the way in solving the world’s biggest challenges...
With the vast store of citation statistics shared across its component databases, the Web of Science stands as the most authoritative compendium of influence and visibility in worldwide research. And now, reflecting new analysis of recent data, the Web of Science anchors an updated presentation of the world’s top-cited researchers...
https://clarivate.com/hcr/worlds-influential-scientific-minds/?utm_campaign=Highly%20Cited%20Researchers_Promotional%20Email_SAR_2017&utm_medium=email&utm_source=Eloqua&utm_campaign=16508-hcrlaunch17&utm_source=eloqua&utm_medium=email&utm_content=report&elqTrackId=b77955e16b30474298aac76cd88f9b3a&elq=1c028d766481459a8349c96b1363c390&elqaid=35309&elqat=1&elqCampaignId=16819
High energy physics database INSPIRE- HEP maintains a list of this sort:
http://inspirehep.net/info/hep/stats/topcites/index
Fraudulent studies are undermining the reliability of systematic reviews – a study of the prevalence of problematic images in preclinical studies of depression
"Systematic reviews are considered by many to constitute the highest level of scientific evidence. A caveat is that the methods used in a systematic review – combining information from multiple studies – are predicated on all of the reports being truthful. Currently, we do not know how frequent fraudulent studies are in systematic reviews, or how they affect the resulting evidence base...
Counter to commonly held beliefs, reports with problematic images were not cited less or published in lower-impact journals, nor were their authors isolated to any specific geographic area. The sheer prevalence of problematic studies, and the fact that we could not find a simple pattern for identifying them, undermines the validity of systematic reviews within our research field. We suspect that this is symptomatic of a broader problem that needs immediate addressing..."
https://www.biorxiv.org/content/10.1101/2024.02.13.580196v1.full
René Spijker has given the best answer.
I second this, by emphasing that selecting studies on the basis of citations may induce biased results, and I can see no sound rationale for this.
If you cannot do a full systematic review due to limited time and resources, consider a mini-systematic review as described here - Article Evidence informing practice: introducing the mini-review
- Badges
- Science topic
- Similar topics
- Geoscience
- Atmospheric Sciences
- Atmosphere