See these related discussions:

https://www.researchgate.net/post/Is_the_peer-review_itself_scientific/1

https://www.researchgate.net/post/Is_peer-review_censorship_or_necessary_evil/1

https://www.researchgate.net/post/What_may_be_the_quality_elements_or_criteria_for_an_excellent_review/1

https://www.researchgate.net/post/How_can_we_obtain_a_better_peer_reviewing/1

https://www.researchgate.net/post/How_do_you_think_we_might_we_improve_peer_review_and_the_quality_of_scientific_publications/1

https://www.researchgate.net/post/How_would_you_improve_the_peer_review_process_Any_ideas_to_make_it_more_robust_and_useful_for_authors/1

https://www.researchgate.net/post/Can_AI_improve_peer_review/1

https://www.researchgate.net/post/If_you_could_change_something_to_improve_how_the_peer-reviewed_process_is_traditionally_done_what_would_you_change/1

Wolfgang R. Dick Much appreciated

Possibly. It's not good if authors are friends with or colleagues of editors, which can happen. I think it would be good if there was anonymity both ways :)

I'd add the process is not truly blind. For most journals, the editor, who decides whether a manuscript is sent out for review and who has the final say on whether a manuscript is accepted, can see who the author(s) of the paper are.

I am no specialist, but this thread seems to assume that anonimity is a must. Many journals prove that this is not necessarily so. In fact, it may be argued that if you do not suspect who the author of a draft is, you may not be the optimal reviewer for it. Likewise, if you do not suspect who your reviewers were, may be the editor made a bad choice or many people declined.

The peer review system is no more broken than the ethics of those who participate in it. I don't think that gatekeeping and quid pro quo are a necessary component of human nature. That does not mean that other approaches cannot be explored, such as crowdsourcing reviews, that may improve or complement double-blind peer review. But discrediting the system by generalizing suspicion, so akin to our Zeitgeist, sounds very much like Lèse-majesté and only burdens those who do a great job, and for nothing.

What is really broken and untenable is not getting any credit or compensation for editing and reviewing. Until we have designed, tested and implemented new ways to filter out bad works and, especially, to help good drafts become even better, peer review is, and will remain, the cornerstone of quality in our collective scientific enterprise. Editors and reviewers need to be supported, not trashed by a system that relies on volunteers.

Ricardo Muñoz Martín This issue is beyond the point of speculation. There are numerous studies showing the existence of publication bias in scientific journals. See just a sample below:

As a cognitive scientist you should be well-acquainted with the myriad of judgment and decisional biases that people (consciously and unconsciously) engage in - reviewers are no exception.

I do agree with you that reviewers should be compensated. However, I also think the decision of whether a scientific work warrants publication should not fall to 1-2 individuals - some of whom may not even be qualified to review the work. There should be a panel of at least 4-5 reviewers to get a more representative and thorough assessment of the quality and impact of the work. And compensation would help make this more tenable.

In the current peer review system, the decision to publish falls on the editors. That many of them will argue it is the reviewers and their hands are tied, or that many reviewers will think they actually take the decision does not change the fact that the responsibility falls on the editor. That is why there is desk-reject, why editors may choose a new reviewer when they detect biases, and even a new round when the reviewers are satified but the editor is not.

That humans have cognitive biases does not mean that we cannot identify and redress or compensate them. In fact, that is how we know such biases exist. If it were not so, then the number of reviewers might be irrelevant, for all of them would be biased, albeit perhaps in different ways. This would also entail that accepted papers are also the product of a bias. So we are doomed, whether the reviewers ar two or twenty-two. I do not think that line can take us very far.

Again, we need to explore other ways to discern the quality of candidate drafts before we trash this one. A large number of the researchers I have consulted about this fear backlashes and unethical power and competition issues. That is a real problem, but it extends way beyond peer review to hiring, promotion, funding, and other areas.

True enough the decision whether to publish falls upon a single editor, which makes the matter all the more alarming - as ultimately, we're in a position where the decision of whether a piece of scientific work meets the scientific standards set by the field is determined by a single individual. I'm afraid I do not share your optimism concerning the impartiality of journal editors (and I'm willing to share a few anecdotes in support of my view).

My point in suggesting that it should be a panel of 4-5 experts who decide whether a paper warrants publication is not that bias would be eliminated entirely, but personal biases will have less of an influence in the publication decision since a group of individuals will vary with respect to experience, opinion, and perspectives in a way that a single individual/editor does not. There would be more room for discussion and reviewers would be more compelled to justify their criticisms or critiques.

One editor making the decision about what constitutes good and impactful science is not at all consistent with the scientific method. In fact, it's directly at odds with it. In general, science strives to remove bias and subjectivity to the fullest extent possible. Nonetheless, nowhere did I suggest the system in place needs to be torn down completely and built anew. My question was how could the issues inherent in the present peer-review process be improved.

to sum up: you think the double blind peer review is a broken system and that it could be solved by enlarging the numbers of people who take decisions to a panel of 4-5 experts, so as to minimize the impact of cognitive biases. You mistrust editors' impartiality and have some anecdotes to support your view. I should agree with you because I am a cognitive scientist and should know about cognitive biases but my problem seems to be that I am optimistic.

I contend that we do have a problem, but it is not in peer review. It is in the ethics of the people involved, a real, deep crisis that spills over to hiring, promotion and funding. I suggest that ending anonimity would make everyone accountable for what they do, that giving credit and compensation to editors and reviewers might dignify the task and also entitle us to ask for responsibilities. I suggest that other strategies such as crowdsourcing reviews should be tried and combined with double blind peer review, but I also know that those of us who are trying some such strategies find resistance in colleagues. That was my answer in a nutshell, my suggestions for improvement.

Usually the most creative cognition/idea work only comes from some certain gifted few who always think about and work out something beyond other people's considerations, and this is exactly what the science most wanted, while the majority of researchers (inclusive of peer-reviewers) not only cannot do it, but also cannot understand it in the first place until the whole discipline has already acknowledged and accepted it.

In that sense, in the interests of science, peer-review should only happen after authors' paper has already been published, in order to liberate scientific creativity, let alone if ethics problems and competent problems related to reviewers are concerned, this will be especially so. More importantly, all comments from all peer-reviewers should also be published online, so that strange comments can be prevented, or, at least, somehow be constrained.