For an evidence synthesis my colleagues and I want to check specific internet pages and databases for relevant information. A comprehensive search would include many unique combinations of search terms. It would be very inefficient to enter all these combinations manually for each website and we would therefore very much like to know about good resources, like tutorials, about methods that would allow us to do this automatically (for exampla by using a script in R or Python)
Hi,
Search Engines Internet search engines are a big part of how we find things online. ... therefore, you need to be able to choose the best combination of key words. Most search engines work best if you provide them with several keywords..
The combination of Embase, MEDLINE, Web of Science Core Collection, and Google Scholar performed best, achieving an overall recall of 98.3 and 100% recall in 72% of systematicreviews. Many systematic reviews use topic-specific bibliographic databases to use the term 'websites' in a broad sense to refer to online resources that .... countries worldwide involves searching a combination of websites..
For more information kindly check:
https://www.mindtools.com/pages/article/internet-searching.htm
Hi Lysanne,
Just some empirical experience from my side for three most commonly used website (i.e. Web of Science, Twitter and Google):
1. Web of Science. Advanced Search: Use the field ID, Boolean operators, parentheses, and search result sets to create a search (http://apps.webofknowledge.com/WOS_AdvancedSearch_input.do?SID=F5LDkY8sH8GTp7LK6Ho&product=WOS&search_mode=AdvancedSearch). I think you might already know this tip, which has been widely used for systematic reviews and meta-analysis. And I also like using “Advanced Search” which is very specific for searching.
2. Twitter. Advanced Search. You can define which word will be (1) exactly contained; (2) exactly excluded; (3) Including any of some words. See: https://twitter.com/search-advanced?lang=en
3. Google. Advanced Search. Same with Twitter that you can define which word can be included. See: https://www.google.com/advanced_search.
And I think you can check whether the website, which you are interested in, has ‘Advance search’, which is very useful for searching information according to difference combinations of search terms.
Hope this information might be useful to you.
Cheers,
Chunlong
I agree with Chunlong Liu. I think we can @use SEO tools for the same and can use multiple combination based on the priority based keywords
Dear all,
Thank you so much for taking the time to rely to my question! I indeed use the advanced search options with Boolean operators for WoS, Scopus & Google Scholar. But I am looking for something to efficiently search a variety of 'simpler' websites without having to go through each website individually and repeatedly (because they only allow very simple keyword combinations). I think the term is 'web scraping'. Any ideas?
Best wishes,
Lysanne
Hi Lysanne, I'm not sure if this is exactly what you are looking for, but we have been using Google’s Custom Search Engine API,
see https://developers.google.com/custom-search/docs/overview for more info.
Ricardo Correia and Ivan Jarić are the ones to approach if you need specific details about the methods.
At this point I'm also taking the opportunity to advertise our preprint here where we deployed these methods :)
Preprint Societal attention toward extinction threats
As an update, for those interested in the answer, this paper may be highly relevant:
Article A rapid method to increase transparency and efficiency in we...
Another possible useful paper for answering this question Article An efficient framework for conducting systematic literature ...
You can use the crawler to map big data if you need it. Python offered a great possibility for that.
- Badges
- Science topic