see Natur, Vol. 517 15 January - World View a personal take on events
Reinhard Werner:
|The focus on bibliometrics makes papers less useful.
Forcing research to fit the mold of high-impact journals weakens it. Hiring decisions should be based on merit, not impact factor.
Science managers and politicians seem especially fond of such ways to assess ‘scientific quality’. But many scientists also accept them, and use them in hiring and funding decisions. They are drawn to the alleged objectivity of bibliometrics. Indeed, one sometimes hears that scientists should be especially ready to apply scientific methods to their own output. However, scientists will also be aware that no good science can be built on bad data, and we are in a unique position to judge the quality of the raw data of bibliometrics, because we generate them through our citation behaviour.
The underlying assumption of bibliometrics is that, by citing, scientists are engaging in an ongoing poll to elect the best-quality academic papers. But we know the real reasons that we cite. Chiefly, it is to refer to results from other people, our own earlier work or a method; to give credit to partial results towards the same goal; to back up some terminology; to provide background reading for less familiar ideas; and sometimes to criticize.
There are less honourable reasons, too: to boost a friend’s citation statistics; to satisfy a potential big-shot referee; and to give the impression that there is a community interested in the topic by stuffing the introduction with irrelevant citations to everybody, often recycled from earlier papers.
None of these citations — for good reasons or bad — express the opinion that the paper in question is a remarkable scientific achievement.
Consequently, highly cited papers often contain popular (but otherwise unimpressive) concepts or methods. If you have a favourite well-cited paper, it is a sobering experience to check 20 random citations. They typically contain little appreciation for the quality of the work.
To be sure, selection for an academic job guided mainly by citation statistics or papers in high-impact journals will get better results than flipping a coin. But it is blind to the difference between someone who creatively develops a research agenda — and is likely to be doing that in ten years — and someone who grinds out papers in a narrow, fashionable subfield.
“When we believe that we will be judged by silly criteria we will adapt and behave in silly ways.”
Many negative effects of bibliometrics come not from using it, but from the anticipation that it will be used. When we believe that we will be judged by silly criteria, we will adapt and behave in silly ways.
A good example is the distortion in the journal landscape — and with it the changes in the style of papers — that arose when the journal impact factor began to be taken seriously as a proxy for reputation.
For example, when Physical Review Letters (PRL) split from Physical Review, it was intended to allow speedier publication of short announcements, which had previously been sent as unrefereed letters to the editor of Physical Review.
It is easier to reach high impact with this format, so the ‘reputation’ shifted from the standard journal to the letters section. Although there is no reason that shorter papers should be scientifically better than long ones, many authors now happily mutilate their work to stay under PRL’s page limit, rendering papers less readable and less useful.
Another example is the way Nature became the top journal for experimental physicists. Life scientists are more numerous and use more citations than physicists, so the impact factors of Science and Nature, which cover all sciences, easily beat that of any non-review physics journal. Despite the higher impact factor, there is no reason why a paper written for a broad audience should be scientifically more valuable than one with an in-depth technical discussion. In fact, in pitching for such an audience, authors often leave out the tricky parts, keep technical terms out of their titles, and overstate their conclusions in broad terms.
What can we do? Simply, individual scientists must resist the trend of making bibliometrics a central plank in their decision-making processes. And we must make this public, perhaps by stating in job adverts that papers will be judged by scientific merit and not by journal impact factor.
Once a hiring decision is made, we should resist the temptation to justify it by quoting the candidate’s bibliometrics to administrators. This reinforces the damaging idea that hiring decisions could be made by administrators in the first place, and makes it harder to justify decisions that do not follow the metrics the next time round.
As the tyranny of bibliometrics tightens its grip, it is having a disastrous effect on the model of science presented to young researchers. For example, a master’s student of mine moved to a renowned research institute for his PhD. Like many institutes, this one boasts of its performance in terms of publications in high-impact journals. So my student was told: “If you cannot write up your research in a form suitable for Nature or Science or Physical Review Letters, don’t bother to even do it.” Such advice, driven by the appeal of metrics to funders, is common but horribly misguided.
If we raise scientists to be driven by such extrinsic motivation alone, then why should they not follow the logic to its natural conclusion, and run off to become well-paid bankers instead?
Nature
517,
245
(15 January 2015)
doi:10.1038/517245a
Only future generations of scientists will be able to say it, but I fear that our "bibliometric-age" will fail to produce the best science!
As soon as I can do so I will attach a pdf of the column, but it seems very relevant to what we are doing here. In particular, Dr. Werner discusses the motivations for citing other research. Of the reasons for doing so, the first paragraph is how I make use of citations. The second paragraph give less usefulness yet seems common enough.
"The underlying assumption of bibliometrics is that, by citing, scientists are engaging in an ongoing poll to elect the best-quality academic papers. But we know the real reasons that we cite. Chiefly, it is to refer to results from other people, our own earlier work or a method; to give credit to partial results towards the same goal; to back up some terminology; to provide background reading for less familiar ideas; and sometimes to criticize.
"There are less honourable reasons, too: to boost a friend’s citation statistics; to satisfy a potential big-shot referee; and to give the impression that there is a community interested in the topic by stuffing the introduction with irrelevant citations to everybody, often recycled from earlier papers."
The following is a url to nature's web site and contains further discussion of the issue.
http://www.nature.com/news/the-focus-on-bibliometrics-makes-papers-less-useful-1.16706
Only future generations of scientists will be able to say it, but I fear that our "bibliometric-age" will fail to produce the best science!
I agree with most of what is said by Dr. Burmeister's comments on the exaggerated importance attached to bibliometric indices as measurement of scientific excellence, and criteria for academic promotion. It is a useful tool, though, for observing the structural characteristics of various disciplines, the divergence and convergence of research fields, social and economical trends, and more.
Dear Avishag,
The point I would like to make is that bibliometrics is a useful tool when used by those aware of its limitations. Information of this sort can be quite useful.
Karl
PS I'm just a 72 year old grad student: nothing more.
the bibliometrics is good method for measure the library equipments such as books and journals citations. but nowadays we are in face of information instead of physical library equipments; therefore we should learn and use the application of informetrics. the old method replace with new methods such as content analysis of documents now. by the study of documents content, we could recieve to good and applicable results in science growth for policy makers of science.
My (personal and of course politically incorrect) understanding is that the emphasis on bibliometrics is just another manifestation of managerialism in science. Managers don't need to understand, and don't want to understand, what the managed personnel do. Managers only want/need a simple way to quantify things without bothering with the details. In the managerial set of mind, a research institute that cranks out a few dozen papers per year and a factory that cranks out a few million gadgets per hour are similar and should be managed in the same way.
Managerialism is also a convenient tool for politicians to quantify what they cannot understand, and use it as a basis for "objective" decisions.
Of course I agree that bibliometrics can be used in good ways. The point is exactly that it isn't.
Bibliometrics is a good tool to measure scientific trends and performance. But it should be applied in a right context with right methodology. Always there has been a controversy when it is applied to evaluate individual’s performance. Indicators such as Impact factor, h-index, g-index, citations etc. have their own limitations. One should be very careful to apply for evaluation. Dr Garfield and many others have warned the use of such indices to evaluate individuals. A year ago, the ASCB’s declaration published against the use of impact factors to evaluate individuals’ performance . Prof. Dr. Richard N. Zare of Stanford University has also appalled at the misuse of citation data for such assessment . The medium of publication and the publishing trends are changing. There is a need to enhance the existing bibliometrics systems to cope up with the changing scenario.
I'm intrigued by these valuable views and was unaware of the misuse of bibliometrics. Perhaps citations are used to authenticate research that reproduces established methods and conclusions and squelch diverse paths in order to perpetuate orthodox ones. On the other hand citations may obviate plagiarism. I'm an unqualified naive medical researcher who has been guilty of misuse without ulterior motives. Just going with the same old flow!
Enrico,
Managerialism has been a problem for decades. During the 80's I was working in the area of military control systems when the first of those little cubic Mac's first came out. Of course the managers bought them, wrote their little basic programs and decided they had software development figured out.
Karl
Knowledge is dynamic and ever changing.It is just a changing pattern of research. Earlier we used to have different methods/tools to evaluate research.These are the new techniques emerged time to time to evaluate the research (Quantitative and Qualitative) Librametrics, Bibliometrics, Scientometrics ,Informetrics, Webometrics ,Altmetrics, etc.This is not the basic research. Basic/Fundamental research will be the same. No body can stop once Creativity.Don't fear for the Bibliometrics it is not devil...
Prof.Sangam, UGC Emeritus Fellow
I think Karl_Burmeister is very right in describing the current state of research & the academic publishing world. I agree that we must be vocal about the situation described in his piece. I am preparing a paper to be presented at the faculty senate at my institution its theme is on "Bibliometrics and faculty evaluation: methodological dilemmas" . I suspect there are some groups which are interested in the adoption of bibliometric measures for faculty evaluation. So my position in the paper is that faculty work is multidimensional and must not be reduced to just bibliometric measures. We must encourage different measures .
For Latin American researchers the situation is worse because our countries confront a poor bibliographic control. Thus, they lacked indexing & abstracting sources and citation indexes. Therefore, in our case the aim to apply sources developed for a culture & language very different than ours to evaluate our faculty performance is disastrous. The tools already in place to gather bibliometrics indicators are not valid, because our literature is underrepresented. Therefore, we in (Latin America) must build our own tools and must not used tools which discriminate other languages and point of views different from those of the so called "mainstream" .
Ketty Rodriguez
Is it really useful to duplicate research methods cited in published studies (in high impact journals) on the genetic determinants of cancer in populations based on the colour of their skin? Does such duplication stem from an attempt to publish in high impact journals and gain funding for the 'racial' comparison?
I think it is fantastic that the subject arouse so much discussion and attention. Many problems that are not visible in formal communication channels are surfaced in this exchange of thoughts.
Regarding the point made by Enrico Savazzi I totally agree with him, specifically, on the potential danger coming from a narrow focus of managers and politicians because, usually these type of persons, are looking for quick fix to complex problems or situations.
Katalin Bikadi ·
Ms Bikadi: I am very excited for your fabulous point, your are very right. We are missing quality because most of the efforts are focused on quantity and high impact journals, disregarding the important contribution .
While I agree that much of the misuse of bibliometric measures is due to managerialism, I would warn against overlooking what we (researchers) do to ourselves. Much misuse of bibliometrics occurs within science without being enforced or even suggested by managers. I know of at least one important funding agency that explicitly banned the use of bibliometric measures in the assessment of project proposals. Officers of the funding agency told me that the scientitst secretly used their smartphones to look up impact factors ... Similar phenomena can be observed in recruitment processes.
The underlying (and very serious) reason is that due to the increasing specialisation in science, a researchers 'areas of competence', i.e. areas in which the content of a publication can be assessed, narrows. Consequently, there is an increasing 'loss of peerness' in peer review - researchers are asked to assess work from fields in which they don't actually work. In order to be able to make judgements at all, they resort to secondary criteria, mostly 'amateur bibliometrics'.
Chapter The Social Construction Of Bibliometric Evaluations
Part of this topic is "reputation" as a thing to be cultivated in ourselves and recognised in others. In a recent issue of Nature (7 May 2015), there is an article by Chris Woolston in the careers section titled "Build a Reputation". For those of us in a hurry, CW provides a summary:
"In 2011, an editorial in PLoS Computational Biology offered "ten simple rules for building and maintaining a scientific reputation. Here is a brief synopsis of the advice.
- Learn to accept criticism gracefully. As tempting as it might seem to fire off a testy e-mail after a real or perceived slight—do not do it. Take time to respond thoughtfully and professionally.
- Do not ignore people below you on the career ladder. It might not be obvious now, but responding to e-mails and phone calls from students can do wonders for your reputation. If you help them to feel valuable, they will value you in return.
- Take publishing seriously. No matter where you are on the list of authors, diligently check everything that carries your name. And you should never accept—or offer—author credits that are not deserved.
- Declare conflicts of interest. Everybody has conflicts, so it is best to stay open and transparent. And if you are ever asked to review a work from a competitor, you should strongly consider opting out.
- Support the scientific community. Review papers, share data, help to organize meetings—in other words, give back to the community as much as you take.
- Do not over-commit. Some scientists fill their slates past the point of reason. With so much to do, few things actually get done well or on time. And that is the sort of thing people notice.
- Be honest with letters of recommendation. You are not doing your reputation—ot the world of science—any favors by writing a glowing recommendation for someone who is not up to the job.
- Maintain your integrity. These days, it is easier than ever to take short cuts with data, images, or text. Even the hint of misconduct can ruin a scientist’s reputation, with good reason. To maintain a reputation as a reliable and respected researcher, you will have to make sure that every aspect of your work adheres to accepted guidelines.”
Chris Woolston
De acuerdo con Karl. Desafortunadamente si no hay análisis de fondo la bilbiometría lleva a aspectos operativos y toma de decisiones desde lo cuantittativo, olvidando la esencia de fondo. Considero que la bibliometr´+ia es importante, pero más allá del número es importante la calidad y la esencia, que se ha ido diluyendo en medio de la norma
If I understand the above (my 5 years of Spanish in New Mexican childhood are less than adequate), As researchers we need to focus on the content of a persons work rather that some superficial numeric measure based on measurables that may or may not be at all relevant.
Ot by my personal logic: Do not believe that technology will evaluate a persons work without our having to understand much about the person. Please focus on content not numeric estimations.
I completely Agree with Karl. While coming to final decision regarding an institution / individual scientist / country profile, many other facors has to be taken into consideration. Merely counting numbers can not reflect on the quality of any research activity, but the senior policy makers some times ignore these facts.
Meager counting articles and citations will not judge the quality of the paper and the author, content is equally important. Bibliometrics is one of the important quantitative evaluative indicators. by Prof.S.L.Sangam
At this point I should use this question to clarify my motivations in using this particular social networking site. As I mentioned in another question (about student's questions) We use questions and answers to titrate all our human relationships and locate like minded individuals.
There are two fundamental goals in my participation in researchers social networking sites. The first is to find collaborators for my research, the second is to perform as a mentor to others, of course not in any ultracrepidarian way.
Karl
Karl,
The analysis you quote from Werner is spot on. We are living in an ultracompetitive age in which those making the decisions are using the wrong methods to make decisions. I would upvote your question and analysis multiple times if this were ethical!
- Badges
- Science topic
- Similar topics
- Academic Journals