We would like to analyze data from Google Scholar - My Citations section where academic data is provided - such as citations, nr of published works, etc. For this reason we need to draw a completely random sample of individuals registered in this database. Do you have any idea, how we can derive such information?
This is how I would do it.
1. I'd use Python, in particular mechanize and Beautiful soup libraries, to build a scraper.
2. I'd use Google Chrome dev tools to see the html structure of Google Scholar result page.
3. I would create a set of scientific topics (not extremely detailed, for example: "fire safety engineering") to search on Google scholar.
4. The scraper would do the following:
a. launch a query on Google scholar for one of the topic you created in 3.
b. access each publication reported as result (mechanize can read all result pages, if more than one is present)
c. look at the authors of the publication and see which one as a link, i.e. is reported in the html in the tag
d. follow the link and go to the author's page on Google scholar
e. gather the information needed on the author's page
f. proceed with the next author
g. if all authors with a link have been examined, proceed to the next publication
h. continue with the next topic on the list
5. Authors are filtered. Those already present are not stored to file.
6. Authors' data are saved .json format, because it's pretty flexible in terms of programs/languages (c++, python, javascript, ...) with which it can be used as well in terms of translation to other formats (for example latex or bibtex). Each author has an ID number, which is an integer value starting from 1.
7. To draw a random sample, I would use a generator of random integers and extract the corresponding authors from the database.
8. Compute some statistics on the extracted database.
There's a bit of coding, but once the code is working I think you can gather data for a number of authors in thousands/tens of thousands. The more queries of scientific topics you're able to create, and the more different they are (try to touch very different areas: physics, engineering, mathematics, psychology, sociology ...) the more random your final database will be.
I hope I've been of some help!
This is how I would do it.
1. I'd use Python, in particular mechanize and Beautiful soup libraries, to build a scraper.
2. I'd use Google Chrome dev tools to see the html structure of Google Scholar result page.
3. I would create a set of scientific topics (not extremely detailed, for example: "fire safety engineering") to search on Google scholar.
4. The scraper would do the following:
a. launch a query on Google scholar for one of the topic you created in 3.
b. access each publication reported as result (mechanize can read all result pages, if more than one is present)
c. look at the authors of the publication and see which one as a link, i.e. is reported in the html in the tag
d. follow the link and go to the author's page on Google scholar
e. gather the information needed on the author's page
f. proceed with the next author
g. if all authors with a link have been examined, proceed to the next publication
h. continue with the next topic on the list
5. Authors are filtered. Those already present are not stored to file.
6. Authors' data are saved .json format, because it's pretty flexible in terms of programs/languages (c++, python, javascript, ...) with which it can be used as well in terms of translation to other formats (for example latex or bibtex). Each author has an ID number, which is an integer value starting from 1.
7. To draw a random sample, I would use a generator of random integers and extract the corresponding authors from the database.
8. Compute some statistics on the extracted database.
There's a bit of coding, but once the code is working I think you can gather data for a number of authors in thousands/tens of thousands. The more queries of scientific topics you're able to create, and the more different they are (try to touch very different areas: physics, engineering, mathematics, psychology, sociology ...) the more random your final database will be.
I hope I've been of some help!
Wow, thank you Luca for your time and effort. It seems that this might be a great approach!
- Badges
- Science topic