Google scholar for systematic reviews: what limit on search returns?

03 March 2014 23 7K Report

Are any researchers who are undertaking systematic reviews also adding a search of google.scholar? And if so, what numerical limit are you putting on results that you inspect? In some earlier trials, I found that scholar returned in the order of at least 10x more results than did the more usual sources (like Medline) which I feel would then artificially distort the number of excluded articles in your flow diagram of articles to be included.

Constantine Kaniklidis Popular answer

From review of this issue I would conclude that:

(1) neither database (Google Scholar (GS), PUBMED) is sufficient for optimal discovery of all highly relevant content in a topical medical search;

(2) that there are several myths and misunderstanding concerning their true differences and divergent focus and specialization;

(3) that there is data to support the contention that Google Scholar can indeed be used effectively for systematic review provided one understands its unique modes of operation and execution;

(4) that one is better served seeing GS and PUBMED more as complimentary than as exclusive of one another; and that

(5) an optimal search process would involve multiple general search databases coupled with specialized collections, with searches being executed best in a highly articulated form (in GS for instance, using scope qualifiers in Boolean expressions). Below I provide the basis for these conclusions:

To begin with, the two search databases, Google Scholar and PUBMED, reflect different relevancy algorithms: PubMed uses algorithms based on MeSH terms, with the most recent articles reported at the top of the list (which likely have not had adequate time to be appropriately cited), while in stark contrast, Google Scholar's proprietary algorithms (first released in 2004 in beta) have been found to favor the number of citations as an important criterion in the initial list of articles, with date of publication not an important criterion [1,2].

GOING HEAD-TO-HEAD: GOOGLE SCHOLAR versus PUBMED

Several studies [3,4] suggest that Google Scholar searches compare favorably with PubMed searches but have both advantages and disadvantages. Another recent study [5], building on these foundations, focused on content relevance and article quality and suggests that the Google Scholar search engine retrieves more relevant, higher quality articles. And in a comparative study as to locating primary literature to answer drug-related questions, no significant differences were identified in the number of target primary literature articles located between the Google Scholar versus PUBMED databases [6]. In addition, as to a single focused query (risk factors for sarcoma) Google Scholar resulted in a higher sensitivity (proportion of relevant articles, meeting the search criteria), compared to PubMed which resulted in a higher specificity (proportion of lower quality articles not meeting the criteria, that are not retrieved) [7].

Similarly, one of the most recent and comprehensive robust studies, this one from the University of Rouen [8], examined explicitly the core question of "Is the coverage of Google Scholar enough to be used alone for systematic reviews", performing a study to assess the coverage of GS specifically for the studies included in systematic reviews, and to evaluate if GS was sensitive enough to be used alone for systematic reviews, in order ultimately to assess the percentage of studies which could have been identified by searching only GS; there were 738 original studies included in a specially constructed gold standard database. The results: GS retrieved all 738 studies (100% hits) mined from 29 systematic reviews, allowing the authors to conclude that "The coverage of GS for the studies included in the systematic reviews is 100%. If the authors of the 29 systematic reviews had used only GS, no reference would have been missed. With some improvement in the research options, to increase its precision, GS could become the leading bibliographic database in medicine and could be used alone for systematic reviews".

What this entails is that despite GS not covering all the medical literature, nonetheless its coverage of the studies of sufficient quality or relevance to be included in a systematic review was complete, so that if the authors of these 29 systematic reviews had relied only on GS, they would have obtained the very same results. In contrast, it's been shown that the recall ratios of Medline RCTs only ranges between 35% and 56% [9,10]. This is in essential agreement with still another recent study [5] where PubMed and Google Scholar searches were compared by evaluating the first 20 articles recovered for four clinical questions for relevance and quality, GS provided more relevant results that PubMed (although the difference was not significant), serving as another reminder that we should not overestimate the precision of PubMed in real life [8]).

LOOKING FORWARD

We await further enhancements to GS to provide reliable advanced search functions, a controlled vocabulary, and improved scope of coverage and currency, but even in the latest instantiation GS performs at a respectable level of recall and precision, and can be enhanced with judicious Boolean expressions and some undocumented qualifiers. All told, as the Rouen study concludes: "the coverage of GS is much higher than previously thought for high quality studies ". (And note that other comparisons have found GS more than credible; in a comparison with Web of Science/WoS [11], the study authors concluded that: "since its inception, GS has shown substantial expansion, and that the majority of recent works indexed in WoS are now also retrievable via GS"). And I would add one further caution: despite the correct claim of many advanced search features being absent from GS but present in PUBMED, nonetheless this has less relevancy that one might believe: only 7% of respondents used these features in their searches for the Canadian study [12], only 37% used controlled vocabularies, and only 20% used filters such as the Clinical Queries feature in PubMed [13,14]. Therefore, in the real-world rather than the theoretical domain, the two search technologies are less far apart than the advanced features suggested, when we look at actual usage patterns.

But the debate will certainly continue for some while, with divergent opinions [15,16]. But it is increasingly clear from a critical review of the data to date that the two databases should be considered complimentary and not mutually exclusive, each with unique advantages and tradeoffs: thus, noting that recent evidence suggests Google Scholar may have closed the gap between itself and PUBMED, and that it is now often leading in searches (with one family of journals reporting that 60% of their traffic is coming from Google Scholar, ahead of PUBMED and other traditional medical databases), University of Utah researchers [17] assessed efficiency and completeness of searching for known moderate and high quality RCTs in PubMed versus Google Scholar, finding that each database consistently identified one of the two highest quality studies, but neither database identified both, yet the difference search time was nearly three-fold (to accomplish the search by experienced researchers, it search time was 63 minutes for GS but 194 minutes for PUBMED, without the later providing any superior results). This again reflects what I have called the INCOMPLETENESS THEOREM OF MEDICAL SEARCH: namely that no single search is sufficient to identify all relevant quality studies, cross-confirmed in still another recent study where Canadian researchers [12] evaluated the recall (proportion of relevant articles found) and precision (ratio of relevant to nonrelevant articles) of searches performed in PubMed and Google Scholar, with primary studies included in the systematic reviews serving as the reference standard for relevant articles, finding that for For quick clinical searches, compared with PubMed, the average search in Google Scholar retrieved twice as many relevant articles (PubMed: 11%; Google Scholar: 22%), with precision being similar in both databases (PubMed: 6%; Google Scholar: 8%). And note that it would be tempting but erroneous to attribute the two-fold greater retrieval manifold as due to differences in content coverage, since 78% of the tested articles were available in BOTH databases.

These and other studies assessing different medical databases have demonstrated that no single search engine provides all the related articles, full capturing of the complete body of available literature on a subject requiring searches over multiple databases, depending on the topic. Thus, a much more comprehensive search would include cross-spectrum searching[18], and as an example I note that I myself use an extensive resource collectivity of approximately 18 databases and tools including ones for specialized content: see "METHODOLOGY FOR THIS REVIEW" below).

As to peer-review, often claimed a major factor that distinguishes Google Scholar (unrestricted) from PUBMED, in fact despite the widely held but erroneous belief that PubMed will only consider Peer Reviewed literature, it is explicitly stated on their website that this is not the case: “Most journals in PubMed are peer-reviewed or refereed. Non-editorial journal-staff review original articles before the articles are accepted for publication. Criteria for peer review and the qualifications of peers or referees vary among publishers. We have no list of peer-reviewed/refereed journals in PubMed; and you cannot limit your search to peer-reviewed journals using PubMed” [http://www.nlm.nih.gov/services/peerrev.html] [19].

USING ARTICULATED/SMART SEARCHES

Finally, it pays to learn how to execute articulated ("smart") searches in GS: thus, in answer to a question in another topic concerning searching for all systematic reviews and meta-analyses concerning HRV (heart rate variability), I advised the Google Scholar smart search (besides a MeSH-enriched PUBMED search):

insubject:"heart rate variability" intitle:("systematic review" | meta-analysis)

or the somewhat more permissive relaxed smart search:

insubject:"heart rate variability" intext:("review" | meta-analysis)

leveraging the power of the scope qualifiers "insubject", "intitle" and "intext" when coupled with appropriate Boolean operators. It also pays to remember that GS is an "opportunistic" search engine, as it will try to data-mine any resources that could be of relevance rather than honing to the more narrow constraints of a formal PUBMED search (often providing riches not otherwise easily uncovered, so that its claimed lesser precision is not at all necessarily a disadvantage, as some of the discovered resources (like dissertations, commissioned monographs, peer-reviewed CMEs, etc.) could themselves - as I have often found - contain bibliographical references to invaluable materials not located through PUBMED, that could greatly enrich the quality of any paper using its technology.

METHODOLOGY OF THE REVIEW

A search of the PUBMED, Cochrane Library / Cochrane Register of Controlled Trials, MEDLINE/MedlinePlus, EMBASE, AMED (Allied and Complimentary Medicine Database), CINAHL (Cumulative Index to Nursing and Allied Health Literature), PsycINFO, ISI Web of Science (WoS), BIOSIS, LILACS (Latin American and Caribbean Health Sciences Literature), ASSIA (Applied Social Sciences Index and Abstracts), SCEH (NHS Evidence Specialist Collection for Ethnicity and Health), and scope-qualified Boolean searches submitted to Google Scholar and SLIM, was conducted without language or date restrictions, and updated again current as of date of publication, with systematic reviews and meta-analyses extracted separately. Search was expanded in parallel to include just-in-time (JIT) medical feed sources as returned from Terkko (provided by the National Library of Health Sciences - Terkko at the University of Helsinki). Unpublished studies were located via contextual search, and relevant dissertations were located via NTLTD (Networked Digital Library of Theses and Dissertations), OpenThesis or Proquest. Sources in languages foreign to this reviewer were translated by language translation software.

REFERENCES

1. Beel, J. & Gipp, B. Google Scholar's ranking algorithm: the impact of citation counts (an empirical study). Proceedings of the 3rd International Conference on Research Challenges in Information Science 2009a, 439–446.

2. Beel, J. & Gipp, B. Google Scholar's Ranking Algorithm: The impact of articles' age (an empirical study). Proceedings of the 6th International Conference on Information Technology: New Generations 2009b, 160–164.

3. Shultz, M. Comparing test searches in PubMed and Google Scholar. JMIA 2007, 95, 442–445.

4. Anders, M. E. & Evans, D. P. Comparison of PubMed and Google Scholar literature searches. Respiratory Care 2010, 55, 578–583.

5. Nourbakhsh E, Nugent R, Wang H, Cevik C, Nugent K. Medical literature searches: a comparison of PubMed and Google Scholar. Health Info Libr J 2012; 29(3):214-22.

6. Freeman MK, Lauderdale SA, Kendrach MG, Woolley TW. Google Scholar versus PubMed in locating primary literature to answer drug-related questions. Ann Pharmacother 2009; 43(3):478-84.

7. Mastrangelo, G. , Fadda, E. , Rossi, C. , Zamprogno, E. , Buja, A. & Cegolon, L. Literature search on risk factors for sarcoma: PubMed and Google Scholar may be complementary sources. BMC Research 2010, 3, 131–134.

8. Gehanno JF, Rollin L, Darmoni S. Is the coverage of Google Scholar enough to be used alone for systematic reviews. BMC Med Inform Decis Mak 2013; 13:7.

9. Türp JC, Schulte J, Antes G: Nearly half of dental randomized controlled trials published in German are not included in Medline. Eur J Oral Sci 2002, 110:405-411.

10. Hopewell S, Clarke M, Lusher A, Lefebvre C, Westby M: A comparison of hand searching versus MEDLINE searching to identify reports of randomized controlled trials. Stat Med 2002, 21:1625-1634.

11. de Winter JCF, Zadpoor AA, Dodou D. The expansion of Google Scholar versus Web of Science: a longitudinal study. Scientometrics 2014; 98(2): 1547-1565.

12. Shariff SZ, Bejaimal SA, Sontrop JM, et al. Retrieving clinical evidence: a comparison of PubMed and Google Scholar for quick clinical searches. J Med Internet Res 2013; 15(8):e164.

13. Shariff SZ, Bejaimal SA, Sontrop JM, Iansavichus AV, Weir MA, Haynes RB, et al. Searching for medical information online: a survey of Canadian nephrologists. J Nephrol 2011;24(6):723-732.

14. Shariff SZ, Sontrop JM, Haynes RB, Iansavichus AV, McKibbon KA, Wilczynski NL, et al. Impact of PubMed search filters on the retrieval of evidence by physicians. CMAJ 2012 Feb 21;184(3):E184-E190.

15. Bramer WM, Giustini D, Kramer BM, Anderson P. The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews. Syst Rev 2013; 2:115.

16. Boeker M, Vach W, Motschall E. Google Scholar as replacement for systematic literature searches: good relative recall and precision are not enough. BMC Med Res Methodol 2013; 13:131.

17. Thiese M, Effiong A, Passey D, Ott U, Hegmann K. Pubmed vs. Google Scholar: A Database Arms Race? BMJ Qual Saf 2013;22:A33.

18. Zheng B, Zheng W, Zhu Y, Guo C, Wu W, Chen C. Are PubMed alone and English literature only enough for a meta-analysis? Ann Oncol 2013; 24(4):1130.

19. Kejariwal D, Mahawar KK. Is Your Journal Indexed in PubMed? Relevance of PubMed in Biomedical Scientific Literature Today. WebmedCentral MISCELLANEOUS 2012;3(3):WMC003159.

Christopher Gale

All I know is that voogle scholar reaches far further. If you look at my citations (I will link mine below) you will find, for instnace, my thesesi, some notes I wrote for teaching and put on a public website, and a lot o conference publications. I tend to use scholar as a secondary ssource, particularly when tracing citations from papers found from public searches. And tI have no magic number.

http://scholar.google.co.nz/citations?hl=en&user=klB8oNwAAAAJ

Ramin Sadeghi

Dear Andrew,

I personally used Google Scholar in selected systematic reviews of mine. However it only gives you the first 1000 citations and not more. It tells you for example your search provide 234543 citations but there would be no link beyond 1000th record.

I personally adopted another way to search google scholar efficiently as a crude search of it can be very time consuming and fruitless.

1- First I extract the included articles I found in Medline and Scopus.

2- I search the extracted articles in Google Scholar and the citing articles (can be retrieved by "cited by" link of the articles) can provide you with many more relevant articles.

This way you can find many more relevant evidence especially non-English ones and as Christopher mentioned theses, etc.

I think this way is a compromise between the high sensitivity (providing much much more articles than other databases) and the poor specificity (providing a huge number of irrelevant articles) of Google Scholar.

Best wishes,

Ramin

Attila Szabo

Recently we did a study comparing resources on Google Scholar and Pub Med. The outcome: The Google scholar listed 126 of the 128 located articles, 69 of these were also listed on the Pub Med, but only 2 (!) were traceable exclusively on Pub Med. So we have concluded that Google Scholar is a more inclusive resource for research on the topic than Pub Med. I personally think, that GS is so sophisticated by now, that any review that excludes it during the search process may suffer very serious limitations. (However, older articles may be present as "citation" only.) If you are interested, I can send you the abstract of the study I mention above, that is being under consideration for a conference presentation. Regards, Attila

Andrew Gilbey

Dear Attila, Thanks for your very useful answer. I would be very interested to read your abstract if you are able to send it to me. Regards, Andrew

Attila Szabo

Dear Farhad!

I send you the abstract as we do not have a full paper! The results are intended for a presentation at an upcoming conference in a specific field,. Therefore, it is not public, yet; so I send it you privately. All I can add, that it was keyword driven, so it was a very specially focused search in a very specific field.

Best regards, Attila

Rose Relevo

Here are the PubMed links to 9 relevant articles about the use of Google Scholar:

http://www.ncbi.nlm.nih.gov/pubmed/17603909?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/19738094?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/20420728?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/22925384?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/23927639?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/23302542?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/24360284?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/24160679?dopt=abstract

http://www.ncbi.nlm.nih.gov/pubmed/24385613?dopt=abstract

In my opinion, Google Scholar is good for exploratory searching, but insufficiently replicable to be part of a systematic review.

I maintain a database of citations for papers about the methodology of systematic reviews and comparative effectiveness reviews. If you would like to join the mailing list to get sent weekly updates, sign up here: http://eepurl.com/PpXJb

Enjoy!

Agner Fog

I have sometimes discovered that controversial articles were not shown on Google Scholar, even though they were shown in other databases. Is there censorship on GS?

Ramin Sadeghi

Dear Agner

Google scholar uses a software to capture the articles. If an article or a publisher does not have the requirements of Google Scholar, it will not be captured.

Ramin

Constantine Kaniklidis