Does anybody know of a way to automatically classify a corpus of about 15'000 articles from academic journals into their respective academic disciplines (e.g. like Journalseek)? Ideally, we would like to (automatically) add a tag with the Academic discipline into say a Zotero database.
Like Caroline says, Web of Science and Scopus each have subject categories. An article in a journal can belong to more than one subject category, so the classification is not discrete. An easy trick is to use an overlay map (Rafols & Leydesdorff) which allows you to see how your articles are distributed among the Web of Science categories. Another method is to take your 15,000 articles, create a similarity matrix using properties in the metadata (we use combinations of title words and cited references) and then cluster the articles, where each cluster represents a niche or specialisation. You can also visualise that (really cool pictures!).
Wow! This Researchgate DOES work!!! Thanks all for those ideas. The articles are, however, from Ebsco (Academic Search Complete). We then put them in Zotero now, including the actual pfds.I thought that would be a convenient way to get all the texts WITH the bibliographical data into a database format , as Zotero sits on top of a straightforward SQLite database. Let me also add that the reason we're doing all of this is because we will process the articles with some textmining tools (e.g. Leximancer), to visualize (semantic!) differences across disciplines, and not JUST statistics. So how associations between certain concepts differ in various disciplines, and not just to count how often they cite each other across disciplines etc.. But so that requires clustering the articles by discipline. Ebsco does have 'subject terms', but these are often too detailed for our purposes. Plus Zotero doesn't save those anyways. SO I guess I'll still have to do some digging here. But yes Edwin, that work you refer too is pretty neat - I was aware of Johan Bollen's work, but not of this. Thanks much! When I figure this out, I will report back. In the meanwhile, if anybody else has any further suggestions , they'll be more than welcome!
Ulrich's periodicals directory (http://ulrichsweb.serialssolutions.com/ ) also has subject listings for journals at title level, but presumably you want something with an API ? I've not used one myself, but the Bibliographic Wilderness blog describes using the WoS and Scopus APIs:
http://bibwild.wordpress.com/2009/04/13/cited-by-from-isi-and-scopus-in-the-link-resolver/
Caroline - I don't feel we're that far along at all. We do have quite a bit of experience with the textmining-bit, but NOT with the 'mapping-articles-to-disciplines'-bit. I have just now finally created an Excel-file, based on JournalSeek's publicly available webpages, of about 50'000 journals mapped to disciplines at two levels (see attached):
* first-level: Arts and Literature; Astronomy/Astrophysics/Space Science; etc.;
* second level: (under Arts and Literature:) Architecture, Art History, Children's Literature, Culture Heritage, etc.; (under Astronomy/Astrophysics/Space Science:) Celestial Mechanics, Cosmology, Extragalactic Astronomy, Interstellar Matter, Meteors, Miscellaneous, etc.); etc. , etc.
But I still have to find a way to map the zotero-entries to the disciplines based on the table I now have.
Generally, I do wish we could go beyond the '2.0'-stuff (based on page rank-type stuff; or citations); towards the 3.0-stuff ('semantic' stuff based on the actual substance of the text, and not just who cites it).
As always, I wish the more 'academically-inclined' folks would have done this type of work already some time ago , but they just haven't. So that means we 'policy-types' have to get into this type of work as well... Sigh!
Well - right now I have the corpus (the articles are essentially about definitions, dimensions and sources of security) and I have the list of journals categorized by discipline. I wish I could either code myself OR have somebody in our team who can OR have the time to learn it (within a few hours - as that is pretty much all the time I have). As I have neither, I soldier on... The irony is that what I am trying to do would probably take somebody with the requisite coding skills a half an hour or so. All that has to be done, is that the 'Publication' field in the Zotero record (Zotero uses a simple sqlite database engine) has to be matched in the Journalseek set, and matched with the appropriate discipline, which should then be added as a tag to that Zotero record. Sigh...
A colleague of ours has now produced a first fully functional version of a Zotero plugin that classifies journal articles by academic discipline. It scans the 'Publication' field of any Zotero library against the 64000 journals that are classified by discipline/subdiscipline in (an embedded copy of) the Journalseek database and then adds a new tag with that discipline/subdiscipline into the Zotero record. If anybody wants to try it out and help us debug it, please let me know.
Just a follow-up. The Zotero plugin we developed proved quite useful for our current purposes (though I would definitely encourage others to improve it, as I am sure there must be more 'elegant' solutions for this). It seems to me that one of the main implications of the increased availability of full-text academic work will be more cross-disciplinary work. That is exactly what we did in the project for which we made this plugin. We searched on EBSCO for all articles that contained the word 'definition' within 5 words of the word 'security', downloaded those (1000s of them) into Zotero, and then sorted them by academic discipline with our plugin. Using both automated (textmining) and human analysis, this allowed use to identify some systematic differences in how various disciplines define security. We also did similar analyses of the ways in which they 'split up' security (dimensions) and especially also the ways in which they try to solve security problems (strategies). And the results were just astounding - I have been in the 'national security' field for my whole life, but I was totally unaware of rich veins of thinking about security in psychology (e.g. attachment security); economics (e.g. tenure security), law (security regulation) etc. I bet many of us will be increasingly interested in systematically comparing concepts and insights across disciplines. And so bibliometric work is undoubtedly great for that (as SNA is a lot easier with things like citation-graphs). But increasingly we will also be able to not just use the citations, but the actual semantic content . Exciting times!!!
For mapping journal articles into specialty categories it is possible to extend the approach suggested by Pudovkin AI; Garfield E. 2002. Algorithmic procedure for finding semantically related journals; JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY Volume: 53 Issue: 13 Pages: 1113-1119. In that paper we mapped journals, but this same approach can be further developed for tagging individual articles. If you are interested in the approach I can write you in more detail and send a couple of relevant papers. Contact me at my email address: [email protected] .
I keep getting requests for this plugin (for some reason by email), so I decided to provide some replies here.
- the plugin is still available on github (https://github.com/ndrsn/zotero-classify-articles)
- it works with a 'fixed' list (as I suggested above) taken from Journalseek in 2011. To be really useful today, an API-based solution would obviously be better
- it was written for Zotero 4 - and should presumably still work on that version. Bear in mind, however, that current Zotero libraries are not backwards compatible). It does not, however, work on Zotero 5, although it could presumably be reworked).
I also still remain interested in this. I find it hard to believe that nothing better would be available today, and would really like to find out. Thanks!
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude