Can we think of giving more credit to the reviewers? - so that it is no longer a totally selfless service. May be in  different ways

1. Journals acknowledging the more vibrant and prompt reviewers.

2. Making 'reviewing record' a standard part of CV of scientists.

3. Journals publishing the number of reviews done by each reviewer - classifying them into different article types. This may be retrievable  from the journal websites, so that claims in CV can be verified.

I think this is why so many writing and editing burdens fall on new scientists (and grad students). Reviewing is a chore, and the only reward is knowing that you're being a good citizen. I've seen the parasites up close and personally, and it's hard to "sanction" them even when detected. What do you do or say when someone is three, four, five rungs up the academic ladder from you?

There are some of us who enjoy editing (I'm one of them), and newly minted PhDs tend to be willing to work harder, longer, and for less tangible reward. I suppose this is one way to keep cycling new ideas and perspectives into the peer-review process. Recruit reviewers from the young and hungry; let the "big-name" professors who don't want to work fade away.

Can we think of giving more credit to the reviewers? - so that it is no longer a totally selfless service. May be in  different ways

1. Journals acknowledging the more vibrant and prompt reviewers.

2. Making 'reviewing record' a standard part of CV of scientists.

3. Journals publishing the number of reviews done by each reviewer - classifying them into different article types. This may be retrievable  from the journal websites, so that claims in CV can be verified.

The rule of thumb of reviewing twice the number of manuscripts as one submits seems pretty widely held as far as I can tell from talking with my colleagues. This anecdotal observation might give us cause for hope that the evolving behaviours, norms and rules about reviewing are an important part of the collaborative, collegial nature of science that many of us hold dear. There are always the few that transgress the rules, for whatever reason. It may be sheer selfishness on their part, lack of time, a sense of self-importance or (and I confess to this myself), the feeling of gloom when faced with the hard slog of attempting to edit the near-uneditable. Regardless, it is worth remembering these 'parasites' are the minority. Is sanction worth the effort? I don't know. Perhaps persuasion is a better way to go.

I know a few colleagues  who will not review manuscripts other than from journals on which they are members of the editorial board, and they rarely reply to requests from other journals. Other colleagues simply haven't thought about the time and opportunity cost to their peers of reviewing their work, or they see the reviewing process as a  tedious waste of time; a sort of of scientific bureaucratic ritual to be avoided and treated with distain. At least one can have a conversation with such people and possibly help them change their minds about why the reciprocity of reviewing is an important obligation for all scientists.

Having just received back, in quick succession, two of my manuscripts that required "major revisions", my initial trepidation in reading the reviewers' comments turned to hope and enthusiasm for the task of revision that lay ahead. All four reviewers had engaged fully, and in a constructive and respectful way. They had provided invaluable commentary that will help me improve the manuscripts. By doing so they have affirmed that what I and my co-authors had written was worthwhile and had merit (subject to revision). I am deeply grateful to these anonymous colleagues for their engagement, wisdom and generosity of spirit. Who would not want to engage as a reviewer when by so doing one can have such positive effects? 

I agree with Matt (re: the rule of thumb) and follow this. I have had a discussion on this recently with colleagues who are closer to the 'big name professor' category you mention and I my sense is that some feel they have done their time in reviewing during early years in their careers, and feel this the onus is on young scholars (I don't agree with this). This is something to do, I think, with the culture of the academy and with publication, and could be addressed in the context of a conference? A panel or side-event on fostering collaborative culture or something, including this issue but also other 'fuzzy' social and cultural issues about publishing--see for example the 'White Bull' paper by Kwok several years ago (2005).

The question of whether or not to review for a journal needs to be seen in it's context. One major problem is that the volume of scientific literature being published is higher than it's ever been and growing. This is partly because new knowledge is being created, but also because scientific publishing is a profitable business. As more and more papers are published and more and more journals are created, big publishing groups make more and more money. In order for their business model to be successful they need high quality professional staff, so they pay proof-readers, typesetters, graphic designers, marketing consultants, (some) editors, web content managers, copy writers, etc etc the list goes on. But they don't pay reviewers. In the old days, science was basically a gentlemen's club and it was considered respectful to ask the opinions of (the few) members of the same club when presenting scientific findings. This evolved into a systematic process of consultation between scientists that we know as peer-review. And as the process has evolved it's become accepted that this service is provided for free. And of course, reviewing obliges us to read and understand other scientists' work, which is good for reviewers as well as journals.

But we've got to the point where peer review has become a bottle-neck because there is just too much work being submitted for this old-fashioned altruistic system to work successfully. Professional scientists are often very busy people, and while some of them are undoubtedly parasites, many of them just need to prioritize and find it difficult to put reviewing at the top of the list. In my own case, I do accept reviews, but I cannot accept all of them, because I like to do a thorough job, and this often means doing some additional research to fully grasp the article's core theme. And because my salary depends 100% on successful submission and execution of research projects, I have to prioritize paid work over other tasks like reviewing. Clearly that's not the case of tenured professors, of course, and since the system has clearly worked for them, it seems fair to ask them to contribute. But I do think we need to think carefully about who is asking us to review, and for what purpose, before accepting. Not all journals are scrupulously concerned purely with the advancement of science and the diffusion of knowledge. Some just care about making money. Do we have a duty to help them in this age of austerity, cuts and job insecurity? Again, I think the answer is “it depends”.

Thanks everyone for your thoughts on this so far. I agree that it would be ideal if 'somebody' created a database that logged all reviews done and kept track of who had done what, so that by doing reviews and editing you could develop some kind of index that would earn you professional credit. But I can also imagine a lot of potential obstacles to getting this implemented. In the shorter term I am wondering whether it might be possible to get journals to set up their databases to find out who has problematic "submission : refused review request" ratios and flag those authors in the system so that handling editors can contact them in person to ask for clarity prior to handling any more of their manuscripts. The problem is, though, that many people probably have very legitimate reasons for not contributing more and may be spreading their broader contribution in other, equally or more important ways (such as running professional societies, advising the public, or just saving the world). Can we find a mid-point between 'it depends' and a simple 'you're being naughty'?

Interesting discussion! 

Personally, I do report to my institution about reviewing and the creation a few years back of www.publons.com, helped me to create and maintain a database of my reviewer's history and to justify of my time spent at reviewing. Once done (I understand that for researchers having 100+ of reviews' history, that could be painful initially), it is very easy to maintain and complete. I use the output (reviewing summary per year etc.) in my reports. I guess for editors, it can be be increasingly used to select reviewers and one could think about instituions requesting a minimum reviewing output from their researchers. We are still far from a "review or perish" policy but it could be tried.

However, I agree with Richard and I don't really think that the reviewing system as it is ("a gentlemen, agreement") is sustainable given the increasing number of submitted manuscripts and journals (even if I really like this altruistic bubble in the research world). Professionalizing the task could be an option even if difficult: researchers being paid by journals (or else?) for a limited period of time (6 months, 1 year) to mainly review (50% or more of their time) could be thought about. This could be seen as another research career step, as taking a sabbatical. Funding this new system would of course be an issue!

Great discussion Graeme, and I've run into this at several journals I edit for. Some journals highlight straight up who regularly says no to reviews (I.e., who isn't bothered with their academic citizenship responsibilities), and so editors get to know who are recalcitrants. You're right, they are often big names. 

I do do like the publons concept, and maybe one day this will be viewed as important by employers (although I hope the same for RG). Payment per review also seems necessary given the profitability of publishing companies. 

I agree that there should be a certain number of reviews performed per paper. I thought this was 5 reviews per paper, so might be going over the top. However I do find reviewing a good way (my only way) to keep abrest of the literature, so it benefits me as well.

cheers

matt

This is certainly a tricky problem, and I doubt that there is a neat solution.

Sanctions against senior researchers who appear not to be pulling their weight will almost certainly also impact their more junior co-authors - who might well be doing more than their fair share for all anyone knows.

As someone who has recently finished his PhD and is postdocing, I do not see many advantages of reviewing (I am willing to be enlightened). Whilst I understand that it may improve my understanding of certain areas of science and may also improve my own paper writing it seems that it is a lot of work for little reward (if the review is done conscientiously).

I also find it slightly nauseating that the publishing companies make a lot of money on the backs of science and free reviewing services. If I thought that these publishing companies had the interest of science at the forefront of their business model then I may feel differently.

I think that payment for reviews should be considered in some form, although I realise that there are issues with this eg bias. I agree that there should be a more formal system to record reviewers so that performing reviews enhances one's CV.

Hello everyone,

Well, it is difficult to answer for 'big-name' professors, but before we consider them as "parasites" and try to find ways of "sanctioning" them, it would be important to understand why they don't have time, or don't take the time to review papers. I believe that there are several, non-mutually exclusive reasons:

1) These academic leaders generally have a huge number of postgraduate students (sometimes 20-50!), and besides lecturing, advising their students, sitting in often useless meetings and doing a lot of admin work which should be done by administrators, they already spend most of their time reading and reviewing their own students' drafts. (This is why they are late or don't deliver at all if they are pushed to review). 

2) Some of them might be using their vast experience to write or edit books which, on top of reason 1), would obviously make it absolutely impossible for them to find time to review additional papers. After possibly 15-30 years of their career spent reviewing papers for others, they allow themselves to be selfish, for once (although e.g. editing a book would be far from being a purely selfish task).

3) Some of them feel that they don't have the required skills anymore (e.g. latest advancements in statistical and molecular techniques) to properly review a paper. (This is why they would underperform if pushed to accept the review).

4) Some of them are "worn out" and/or bored by this type of activity, or have just reached another phase in their life, with different goals. (This is why they would do a lazy job if pushed to accept the review).

5) Some of them have so much experience and such an amazing CV that reviewing papers would not bring them any benefit anymore.

So, all in all, points 1 and 2 provide more-than-reasonable excuses, point 3 corresponds to an ethical and paradoxically professional conduct, point 4 is a "nobody is perfect" scenario, and point 5 is a selfish attitude. It could be considered as a "parasitic" approach if that same individual expects people to review his/her papers while doing nothing in return, which I agree, is likely to happen/be the case.

There are obviously different ways in which one can assist in the advancement of science, and different times at which one can be of real value in any of these tasks. Individuals belonging to all five categories may well have contributed to their fair share of reviewing work, and in some cases pushing them to do so would be unproductive (delays in reviews, no constructive comments or no comments at all, inability to appropriately evaluate the scientific merit of papers). So, with the exception of a few "selfish" late career scientists (category 5), I don't know whether this would be worth using the time and energy to developing a "sanctioning system"?!

My gut-feeling is that there are more "parasites" (but I might be wrong) among the mid-career scientists, and this is essentially driven by the fact that science has now turned into a form of business. We are now put under so much pressure to publish in order to get funding that time used to review papers equals time wasted in writing one's own papers - some might even see it as assisting others at moving forward and increasing these same others' chances of obtaining funding they are also competing for. I disagree with the idea that "big-name" professors create a large amount of work for editors. A large amount of innovative research is driven by the desire of PhD students and post-doctoral researchers to "make their own mark" in the world of science, of course under the guidance of a mentor. I therefore consider it as their duty, once they have published a substantial amount of papers and acquired enough experience, to take over reviewing duties. In addition, as mentioned above, these early and mid-career scientists are generally better informed about the latest techniques in their field (with some exceptions), and would therefore be the best possible reviewers. Depending on the topics, of course, the input and assistance of more experienced researchers (this includes the "big-name" professors) would be needed.

I agree with Richard that we are currently facing huge difficulties due to the increasing amount of scientific journals and papers submitted every year. We have reached a level where everything is about business, and considering that scientists have been abused for too long, it is now time to have journals paying reviewers for their services. You would be surprised at how responsive people would be after receiving a reviewing invitation! And even the idea/goal of improving one's CV would become a secondary one. Everyone involved in the publishing business is paid for their services. Only scientists, who do in fact 90-99% of the job (including reviewing), don't get paid. It seems that being a scientist is too much of an "honourable" profession to deserve any financial retribution. Worse, some scientists now accept to pay journals huge amounts of money to have their papers published, while some excellent "free" alternatives (acclaimed journals with similar IF) exist. To me, this is not much different than having a prostitute paying her/his clients to have sex. At least this is food for thought. In this way, I feel as nauseated as Rory is!

To conclude on a more positive note, and to concur with Matt C and Matt H, reviewing papers is not only a collegial attitude but also a very enriching process, and sometimes correspond to the best and/or only way to stay abreast with the literature in one's field. In other words, we have our work cut out by esteemed and hard-working colleagues who collected and reviewed the latest literature, so the least that we could do is to assist them the best we can in professionally suggesting some possible improvements. On the other hand, authors should acknowledge reviewers for their - sometime invaluable - input, which is not always the case... (and possibly constitutes another topic for discussion!). The crucial point is to find a balance between writing and reviewing papers, something in which I have miserably failed over the past years (with a bias towards reviewing).

Finally, it is important to remind the scientific community that a vast majority of scientists are not native English speakers (and in fact writers), and whereas there is no scope here at discussing why we should use English as the scientific language, it would be important to recognize that some colleagues need to make many more efforts than others to produce a/the same scientific output. If competition for funding ought to be fair, and if scientists are the real "gentlemen/women" they claim to be, then another collegial attitude would be to assist non-native writers to improve their manuscripts from time to time - yes, they are scientific editing services for that, but they are costly, and this is money that could be used to fund research instead. Irrespective of this, as a non-native English speaker, reviewing activities have assisted me in tremendously improving my writing skills. Still a long way to go, but I feel that it's much better than before!

Regards

Emmanuel

Very interesting debate.  I realise that, at an average of five manuscripts per month and at least one edititing per month, I may have been stretching myself. I rarely turn down review invitations from journals where I am an editorial member, and review invitations from credible journals as long as the submission is on something in my field.  I, however, have serious reservations with review invitations from unscrupulous and parasitic journals. 

I agree that we need to create some incentives for reviewers, but I also think that seasoned researchers need to take reviewing as some form of 'corporate responsibility'.

As most of you have said, reviewing helps me to keep abreast with my developments in my field. 

Voster 

As an associate editor myself, I encounter this problem frequently, and I think it is an issue because many academic institutions do not acknowledge the work that people put into the peer review system. This needs to change or the entire process will be undermined. Unless written policies are in place to deal with people who refuse to review papers, I don't think we are justified in taking action. Definitely room for debate and it will be interesting to see if more journals take a stand on this issue in the future.

As much as some of us might not like the idea, I think payment for review is the only long-term solution (as alluded to already by several people on this thread). The publishing houses make large amounts of money off of our free labour, and we often, and increasingly, even have to pay them to publish our work. Why should we be expected to enrich them for free?

I think that a relatively small amount (say $100 per review) can be incorporated into already existing publication costs and handed down to the reviewer. Editors can then decide whether a reviewer deserves payment based on the quality and punctuality of the review, and choose not to use the reviewer again if they feel their review was below par. Perhaps reviewing will even become competitive if reviewers are compensated for their work. This is how almost every other field works (would a mechanic or lawyer advise their colleagues (and competitors) for free? Can you ever receive free advice from a medical doctor or business consultant?

I know that as a postdoc, a $100 (or less) payment for reviewing would be a big incentive to produce more high quality reviews in good time. I already review multiple papers, often at the cost of my own work, and this while trying to establish a career in a highly competitive industry. Some small payment would also help the often less than desirable financial state of PhD students and postdocs.

Following on from Andrew's suggestion above, I note that very few scientific journals offer financial incentives to reviewers, but they could be encouraged to do a lot more. Fewer and fewer of us can afford the charges of the Big Five science publishers - Reed-Elsevier, Wiley-Blackwell, Taylor & Francis, Sage and Springer. (For an introduction to basic facts and figures for this group, see http://www.vocativ.com/culture/science/five-corporations-control-academic-publishing/.) Journal subscriptions, online access, page charges and reproduction costs for colour figures are increasingly taking up a larger proportion of the shrinking operating budgets of researchers. Offering significant incentives for a review by way of reductions or waivers on publication costs could help partly redress the questionable ethics of the publishers' business model (whereby as researchers we wind up with little choice but to pay them, typically via taxpayer funding from public grants, to allocate academic capital). An alternative argument is that such an viewpoint is simply colluding with the privatisation of public knowledge at increasing cost (see, e.g. http://thecostofknowledge.com/). I am moderately confident the message would start to get through if more of us filled in the box in the review request for 'reasons for declining to review' with a statement that we would be glad to review the manuscript in return for a discount on page charges, but until such time as in-kind payment for our time and expertise can be made we must politely decline.

I review a lot for Elsevier, partly becausse for each paper I get a month's free access to Science Direct/Scopus and full text access to Elsevier Journals. Since I work at an NGO with no budget for the library facilities that universities have, that makes it well worthwhile to review papers that I would otherwise decline.

Given the scale and economics of internet literature access, I get something valuable to me and Elsevier gets something they need at no incremental cost to them. Win-win and very easy for other publishers to implement. For university academics with access to a library the benefits might be smaller, and Matt Colloff's suggestion of a discount on page charges is an excellent one.

Matt's idea on reasons to decline is also something that we can easily apply - especially when in two minds as to accept an invitation to review.

Hi , very interesting discussion. I few months ago I received a mail with information about a web page that want a professionalization of peer review, aiming a payment for per review: https://scirev.sc/register/?paidpeer

What do you think about it?

Lucas, I have used SciRev many times over the last few years, not as a paid reviewer, but for the journal reviews - reading them and contributing my own. I recommend as many people as possible join so that more data on journal handling times and their review process is made available. 

I also signed up as a paid reviewer some time ago, but have never received a request. With free reviews so easily accessible, the publishing houses probably see no need to pay reviewers (yet).

I don't know about this particular one, but I signed up for something similar early this year and so far it has generated exactly zero review requests and zero income.

Why would a journal pay for a review when they can get one for nothing ?

Thanks everyone for your interesting comments and thoughts. It seems to me that we are trapped in a social dilemma situation here, with individual and group benefits at odds with each other and publishers profiting from the outcome. In response to Lucas, and picking up on other voices as well, I can see the potential benefits of a paid review system but I also worry about its potential consequences for quality, partiality, and the screening process. I would for example expect publishers to restrict how many paid reviews editors could solicit in a year, which would drive 'instant rejections' up and might work against papers that are more creative or unusual (or editors who are more willing to 'let the people decide'). Depending on how the system were set up, reviewers might also try to either minimise review time by writing very soft reviews or write unnecessarily long, detailed reviews in order to trigger higher ratings. Are we ready for the rise of the professional reviewer, who doesn't practice science actively or have any field experience, but gets to set our agenda? I also keep Campbell's law in mind, because the commercialisation of reviews and/or increases in recognition would quickly lead to review number and quality being used as indicators by university administrators. Campbell's law states that "The more any quantitative social indicator is used for social decision making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Having said this I suppose that many of the highest impact journals already use a form of paid review system by hiring professional editors who do a lot of screening and discussion before sending an article out for review.

I recently received a mail divulging " Reviewercredits.com" which addresses Asokan's post on giving more credit to reviewers.  I haven't had time to check the site out yet. Does anyone know about it and if it is legitimate?

http://reviewercredits.com/?utm_campaign=website&utm_source=sendgrid.com&utm_medium=email&utm_source=All+reviewers&utm_campaign=f2cc71a0f8-Fifth_Launch8_24_2016&utm_medium=email&utm_term=0_c9975c3970-f2cc71a0f8-122776365

When I wrote about giving credit to reviewers, I really didn't know that such a system existed. This is probably what happens in this digital world - by the time you think of some 'new' idea, someone has already put it into practice. The concept seems attractive. By avoiding financial rewards, 'corruption' possibilities would be less, one may presume. It is logical that one gets 'scientific rewards' for 'scientific work'. 

It is early days to tell whether this system will be popular with reviewers and editors.

In response to Samantha's message:

I have checked the link and it looks like a great idea, and I might end up being one of the persons claiming reviews on this site, if this is legitimate (at first sight there is no reason why it would not be). This would certainly motivate some people who generally reject reviewing invitations to accept the job, although some of the same potential downsides as listed by Graeme for paid services can be expected - notably quick, soft reviews to increase the number of reviews that can be taken over and to accumulate reviewing credits asap.

However, I disagree with the website creators that "This is where the hard work of peer-review is, finally, rewarded!". The hard work would not be rewarded, it would just be put at the forefront, and get "officially" accredited in case reviewing activities really matter on CVs. In addition, so far, I have never heard of anyone being put in doubt regarding his/her list of reviews - although some individuals probably beef up their CVs when applying for jobs. The non-financial reward for reviewers is not for for them to let colleagues know that they have reviewed substantially (as indicated by the reviewing credits), but rather to know that:

1) if they have been asked to review a paper, then they must be given some credit by the scientific community;

2) they have assisted colleagues in improving their work;

3) they have assisted in getting science moving forward; and

4) they have learned themselves something in the process.

Point 1) is obviously not systematically correct, especially when editors choose reviewers suggested by authors, and these are selected because they might be more "favourable" or "lenient" in their review. Point 2) gets even more rewarding when authors acknowledge reviewers, which is by far not always the case.

The problem with the reviewer credit system is that it will be even more favourable to publishers, as scientists will be even more inclined to work for free, hence supporting a system which is not too far from "modern slavery". Why should it be that people who work overtime and on week-ends (indirectly to ensure that other people get a job and a salary) don't get paid for their efforts? Just because they are passionate and want to get science/knowledge moving forward?

Cheers

Emmanuel

It is a good idea. I'll take a look at it soon.

Some publishing houses, such as Elsevier also recognise reviewers' efforts

". If a reviewer has completed at least one review in two years, they become a ‘Recognized Reviewer’. ‘Outstanding Reviewer’ status is awarded to those who belong to the top 10th percentile in terms of the number of completed reviews for a specific journal in two years."

https://www.elsevier.com/reviewers/becoming-a-reviewer-how-and-why#recognizing

Dear Graeme,

I feel that you should not mix up the two issues at all. There can be many reasons why 'a high ranked professor' does not perform reviews for the journal to which he submits. I agree however, that anyone who does not respond, does do a disservice to the scientific community.

Best, Matthias

I have been for several years an Associate Editor for an Elsevier Journal.

I always found very difficult to involve someone to review the manuscripts, for many aspects.

First, the problem was the quality of the reviews and also the time (generally 1 month) to have back the required review. The latter is very short for most people but to the contrary it is very important to publish papers in a short time as possible.

For my job of Associate Editor, which implied a considerable effort in time, I was very little paid in money, but Elsevier, in addition, provided me a full access to http://www.sciencedirect.com/ which was for me very interesting.

In the same time Elsevier, in order to make more collaborative the potential referees, gave the referees in return for possible reviews, the full access to http://www.sciencedirect.com/ for a month, for each reviewed paper. I think it was a good idea!

I don’t know now the policy of Elsevier about this questions.