For Excel, in order to use the data, the data should be in an acceptable format with unique identifiers such as a "tab" or "comma" (the file for excel is called *.csv which stands for Comma Seperated Values). You may be able to do it more efficiently if you are good at SQL and with a bit of programing experience, but the easy suggestion would be to use MS Word. Copy - Paste the data and sort it in Descending order. You would anyhow have to manually delete the duplicates whether it is Excel or Word.

Good luck.

Funny enough, a simple Google Search provides you with plenty of examples of how to extract Google results into CSV format which can be imported into Excel:

http://www.searchenginejournal.com/export-google-search-results-to-excel/11253/

http://www.marketing2oh.com/scrape-serps-for-seo-analysis/

http://jurnsearch.wordpress.com/2012/01/27/how-to-extract-google-search-results-with-url-title-and-snippet-in-a-csv-file/

Don't be put off by the fact that these articles primarily target web developers and search engine optimisers.

For me MozBar didn't work. And SEOquake did the job only on google, not on scholar. For now Outwit Hub seems the most appropriate one, but in the free version lots of features are missing....

I was looking for a method to do something similar. I found two workable solutions.

The best was to use the Publish or Perish software (http://www.harzing.com/pop.htm). It cycles through the pages of a Google Scholar search results list and copies the basic information for each result to a results list that can be copied in CSV or Excel format.

The other method was to use Zotero (www.zotero.org) as an add-on to Firefox. You can use it to save the results on a single page of a Google Scholar results list (maximum 20 items). The drawback of this is that there doesn't seem to be a way to cycle automatically through the entire results list.

@Daryl Grenz 

Thank you! Publish or Perish works perfectly for me.

but publish or Perish works only elsevier!

@ Daryl Grenz

Thank you! Publish or Perish work for me.

Publish or Perish...excellent.

My problem is that I am conducting Google searches of topics, defined by complex boolean search phrases. Publish or Perish does not handle that, so I am back in square one.

Giora

You could also use the Mendeley webimporter (https://www.mendeley.com/reference-management/web-importer)...

Thank yoyThomas. I have no experience with Mendeley but will try it.

Giora

I am not sure if Mendeley can help in downloading list of papers to be further used for bibliometrics and SLRs. Mendeley allows you add one paper at a time, I think.

Giora, have you tried  "All of the words" field and checked "*Title words only"? I think it would help you even if you are using boolean search strings.

Otherwise, consider replacing phrases with words for example, "information security management" OR "information security awareness" can be replaced by ("information security" AND ("management" OR awareness). Then "All of the words" field can be used.

Ali

 Thanks a lot, but this is what I have done. The problem is that I have to use several different variations (otherwise it is too long), and each search yields many duplicaions. Therefore I must have a list of all of them which will allow identifying and deleting duplicates.

Giora

Giora,

I can understand as I have been in the same situation as yours. If I have rightly understood, you have to use several variants of search string as the search keywords are too many; and you are not sure how to remove duplicates in this regard.

If you are able to divide you search string into several smaller set of search strings, you can conduct search for every string separately. Download them into Excel using PoP, merge them and use "Remove duplicate" option in Excel. You will get unique records after that. Kindly be advised, PoP can only search 1000 records at a time and thus if you are going to have more than 1000 records within a particular timeline, you won't be able to see them. An alternative is to divide you searches into smaller timelines, for example. Also, due to repeated PoP queries to Google Scholar's servers, the server may lock you out for certain period of time.

If nothing works, think of Web of Science or Scopus, for example. Both them have better "quality coverage" as compared to Google Scholar. Of course, citation scores from these two databases are less than Google Scholar but that will not make any difference if your aim is to conduct a literature review.

Ali

 Thank you. But how can I transfer the Google search results to PoP?

Giora

You will simply query PoP to execute the search string query. PoP automatically searches Google Scholar and lists the results in PoP which are then downloadable into CSV, TXT, XLS formats. PoP can also search in Microsoft Academic database, in case you are interested.

Daryl has given the link to the PoP's website and there is a very short tutorial on how to use it as well; may take 15-30 minutes to get yourself familiarize with PoP.

I am unable to use Publish or Perish. I have used it earlier but now it is showing the following error:

"The data is invalid".

Please help.

Publish or Perish...excellent.

Publish or Perish is great

Publish or Perish works fine for me

Publish or perish as well as Zotero offers great services in this.

I have just downloaded publish or perish for importing references for meta analysis but I cannot understand how to use it. Is there any tutorial for this purpose?

Bibi Kulsoom, there is a tutorial available on PoP's website. Moreover, a quick search on youtube will also handy.

Thanks Ali Farooq, but all the tutorials are about finding H index of authors and no one is talking about exporting search results from Google Scholar or Pubmed. can you or some one else give me the link of the tutorial? It will help.

Bibi Kulsoom, on this page you will find the instructions on how to use PoP for general search:

https://harzing.com/resources/publish-or-perish/manual/using

I have not checked if PoP can extract results from Pubmed.

Moreover, if you not sure how to copy results from PoP to excel sheet; Once the search is complete you see "Copy" button to the right. By clicking on the drop down menu button you will find all kind of options (in which form you would like to export your results).

Hope it is helpful!

Thanks for your help Ali Farooq. I am currently searching through Publish or Perish by using the instruction given website you recommended. I am totally new to this software. I am not clear should I click "Save as the full query report" OR "copy to the clip board and full query report"? How this will save the seasrch results for my meta analysis efficiently?

As I said it depends upon you need. If you wish to go through the extracted results using any software such as Mendeley or Refwork (select Results as Bibtex), in case Endnote or RefManager you can export the file in the respective format.

What I usually do is to export results to Excel with header. Once you click this option, the whole list with headers will be copied to the clip board. Now open the Excel sheet and simply "paste" or Ctrl+V. All the data will be copied to Excel sheet. Now you can save this sheet and do further analysis.

Can I search Pubmed through Publish or Perish? the option of Pubmed search is not visible in the software.

Thanks Ali Farooq. I am trying to get the results according to your way.

Ali Farooq thanks it is working. Can you guide about Pubmed and Scopus search through this software?

Thanks that was useful !

Bibi Kulsoom, I haven't use PoP for Pubmed and Scopus. However, you can download meta data from Scopus directly.

Thanks that was useful !

Have you tried https://scrapinghub.com/ ? or other:

https://www.nature.com/articles/d41586-018-04190-5

https://academia.stackexchange.com/questions/34970/how-to-get-permission-from-google-to-use-google-scholar-data-if-needed

https://stackoverflow.com/questions/tagged/google-scholar

https://stackoverflow.com/questions/13200709/extract-google-scholar-results-using-python-or-r

https://github.com/ckreibich/scholar.py

https://github.com/OrganicIrradiation/scholarly

https://cran.r-project.org/web/packages/scholar/scholar.pdf

https://libraries.mit.edu/scholarly/publishing/apis-for-scholarly-resources/

https://harzing.com/resources/publish-or-perish/manual/reference/dialogs/scopus-api-key

Indirectly YES. First import to CSV format then to MsExcel format

Thanks you so much very intersting contribution

The best solution is to use the Publish or Perish software .Download the data from Google Scholar and save it in a RIS file. You can further analyse the data by using VOSviewer for and creating maps.

Publish or perish *does* have results limitations. Be mindful that it may not be exhaustive if you exceed them. If someone knows whether the API access is able to exceed that results limitation, I would love to know about it and how to implement.

Also note that with Scholar searches in Publish or Perish, they will be returning the highest cited/ most relevance results in any truncated list of results... (e.g., top 200)... not quite the same as how other databases search and return hits.

Rightly pointed out. @Robert!

Razieh: In csv you can se the title column. Isn't that what you are looking for?

Razieh,

I think there is an option where you can copy the results with header information, both for csv and excel format. Use that instead of simple copy option.

I am glad that I was of some help. Cheers @Razieh

Very nice Daryl Grenz . I am yet to see how I can do it from google scholar directly without downloading any browser extension. For my own searches, I can get the download option in csv format that can be opened in excel. But searching publications o ther than mine and saving directly into csv is something I am looking forward to if someone can help.

Nice!! That is really what i need for literature survey!! Thank you!!

thanks for sharing

Thanks for sharing. Unfortunately the PoP can only extract 1000 citations from GS and 200 citation from Scopus. I am conducting a systematic review and want to search engines such as BASE, AGORA, JSTOR, etc. Do you know any citation scraper that will do the job?

Thanks for sharing! In addition @ Konoutan Médard Kafoutchoni have you ever used Rayyan QCRI to manage you downloaed CSV articles say n= 2000 papers.

You can use this application to screen down the tittles with include/exclude undecided and duplicates removal options. This is so powerful tool for systematic review

Try it link https://rayyan.qcri.org/users/sign_in

Gaspary

Hey, @Daryl Grenz the http://www.harzing.com/pop.htm worked perfectly alright for me. But is there a way I can configure to also download "abstracts from the selected papers?

Publish or Perish really useful tool for researcher. thanks you for introducing here.

PoP is the best. Thank you Giora Rahav

Hello Everyone. I already installed Publish or Perish, and I performed a research on Google Scholar. Then I tried to Save as CSV. But all the items are kind of messy in the file. I would be grateful with some advice

google scholar now truncates the outputs so Publish or Perish requires a work around to complete. Caveat that the max number of outputs is 1000. For my use case, an error message arose which maxed mine out at 980. Still useful, because it outputs year, citations, citations per year, author list, title, journal, volume, link, abstract. Also keep in mind that any non-standard letters in the author names screws up the output to varying degrees... need to manually repair those. still more efficient than manually compiling 980 references into a .csv though.

i used the Mendeley work around listed on the site (https://harzing.com/resources/publish-or-perish/tutorial/google-scholar/truncation). *The key difference from the posted link is to output BOTH the .csv and BibTex file from PoP for later use*.

Then, use this workaround to convert the Mendeley BibTex output to a csv using JabRef (https://cynthiadangelo.com/2012/10/22/exporting-mendeley-tags/). This repairs the truncation but loses the metrics.

So the final step requires a manual matching of the original PoP .csv output containing the metrics (citation etc) to the "fixed" (non-truncated) output from Mendely-->JabRef. The rows pretty much already match up so shouldnt be too much of a problem to do manually, but i wrote a python script that does this if anyone is interested feel free to DM me.

How can I perform a search on Google Scholar using Publish or Perish, but limit it to a specific website?

On Google Scholar I just add "site:" on my query, but on PP there is now search field to use this filter.

Dear Tobias Nyumba

How do we download the papers if there are more than 1000 papers through harzing? Do you have any idea?

Hi Rajeev Shrestha , I delimited by year. It helps when you have less than 1000 articles per year.

I did a search in google scholar using publish and perish. I got 16k+ search results. As suggested, I tried the search year wise but the recent years had more that 1k results. Is there a mechanism to save these many results in publish and perish?

if you have an OR statement in your string, you can split it for these years

Dear Felipe André Zeiser , I don't understand your last answer. I'm doing a search and when I filter for the year 2020 I have 1180 articles. Could you give me some advice on how I can download the 1180 articles in the software?

Hi,

So when we have more than 1000 articles per year, there is no way to get all the articles. I suggest that you perhaps deselect the options for citations and patents in the search. If you are only looking for scientific articles, you can also remove books from the search using the following command along with the string: '-books'

About Google Scholar:

Article Which Academic Search Systems are Suitable for Systematic Re...

If you just need the number of results (without getting the references), then one easy way is using SEOquake in Chrome (or added into your favourite browser).

If you need the references and publication details, then use the Publish or Perish software (http://www.harzing.com/pop.htm), knowing that you will not be able to retrieve more than 1,000 results per search. Also, after a few rounds, Google may block your IP for 24 hours. One way to go around is limiting the results (e.g., by limiting the dates) and also by using different computers.

Using R language

You want more than 1000 articles from publish or perish? Say you want articles from 2010 - 2015. You can restrict search to 1 year. Eg 2010-2010 then download results. Then change to 2011-2011 then download. Continue until 2015.

It takes some time but you can get all your articles on Excel. I just tried it.