Here is a recent special issue of Frontiers on open evaluation; http://www.frontiersin.org/Computational_Neuroscience/researchtopics/Beyond_open_access_visions_for/137 Many contributions discuss advantages and disadvantages of an open peer-review process, and Alexander Walther and me contributed a proposal for a framework of open evaulation: http://www.frontiersin.org/computational_neuroscience/10.3389/fncom.2012.00032/full

I think that as people gradually start to communicate about science outside of the traditional channels, open peer review systems will see more use.

unfortunately, I cannot get access to Jasper´s article. the link he provided only gives me access to a one-page summary. the option to "buy the ebook" does not work - on my computer anyway. It would be interesting to read the article. However, without the insight he has added to the debate, I am forced to maintain my view that the peer review process should remain "secret". It allows the reviewer a certain freedom to say things that they might be reluctant to say if they were clearly identified. with computer social networks, Helia and I could be "network" friends, and the publisher without knowing that relationship sends her article to me to peer review. The peer review process assumes that the publisher has selected the best people in the field to conduct the peer review. That may not always be true. Now, Helia should understand that the review process is not as "secret" as it appears. A review that comes back: "the author should have consulted the works of Smith; Jones and Smith; and Gómez, Brown and Smith", for example, was with a 75+% possibility, written by Smith. If Helia is writing in Portugal about some aspect of Portuguese influence in the New World during the colonial period, for example, and the peer reviewer is in Brazil, the reviewer might say "the author is really not up-to-date on the latest Brazilian publications". The reviewer cites them (none of them are available to Helia in Portugal or they have not hit the Internet yet), Helia might ask herself -- who is in Brazil, dealing with Portuguese influence there, who would know all that.? It would occur to her ---- oh, i know who that is. Tony is doing that work. Again, 75+% possibility Tony wrote the peer review. There are other tricks to narrowing down who did the peer review. until the world gets to read what Jasper wrote and gets around to accepting it, this is what we have to live with. Keep writing Helia.

Hi Tony, thank you for your insights. Sorry for the bad link. You can find the pdf here on my profile: https://www.researchgate.net/publication/228106294_FOSE_a_framework_for_open_science_evaluation

Alexander and I think that both anonymous and signed reviews have their merits. "Open" in this regard then refers more to the fact that the reviews can be read by third parties, and that an unlimited number of reviewers may participate, be it anonymous or not.

Article FOSE: a framework for open science evaluation

WOW, Jasper. that is quite the article. I need to read it in more detail and get back to you. I think what Helia is talking about is the "secretive" nature of the pre-publication peer review process. but we need to confirm that. i am thinking: She sends an article to a journal for consideration. the journal sends it to certain "experts". the "experts" respond with an anonymous review. there are, depending on the journal, strict peer review guidelines. I think that in your article there is a comment about reviewers not following the guidelines. Having done peer reviews, it is a time-consuming process, if you want to do it right. i And the peer reviewer has to be open to new, sound ideas. The journal publishers tend to think that the "experts" are just sitting there waiting to be asked to do do a peer review and that every day is a good day for the reviewer. NOT !!!! . let´s wait to hear from Helia before moving on to the next stage.

Hello Jasper and Tony! Sorry for taking so long to answer...

Yes Tony, indeed I was talking about that "secretive" nature of the process and how we are all losing some positive aspects of the open process. And that goes for the blinded/double blinded review process that is used for journals as well as for conferences.

For example: I have submitted a paper to a conference and got three very different reviews. Two were positive and had constructive critiques that were actually useful for improving my work. The other one was severe, very unfriendly, and I even got the impression that the reviewer didn’t bother to read properly some parts of the text. That was my first wakeup call – I’m new at this, you see… :)

But, as I was organizing a small peer reviewed conference, I wondered if we could have an open system. I was thinking that there must be a very good reason for the persistence of the blind reviewing process, especially in social sciences. At a first glance, I thought that the open process would let everyone involved learn from each other: as an author I can continue to discuss some ideas with a particular reviewer at the conference or understand how his/her background research influence how he/she envisages my work; as a reviewer I can learn how others (possibly experts) see and judge the same work and compare it with my personal view.

I took a glance at the FOSE framework proposed by Jasper, and it looks like a leap into the future! How has your community been accepting it?

Hi Helia and Tony, thanks for your feedback! We are currently testing the FOSE framework on http://fose1.org/ , Once we'll have a fully functional beta version we'll invite people to test it. We are also planning to ask for funding. We currently think the main challenge will be to encourage scientists to take part in an open review process. Anyone who is interested is welcome to join us in developing this initiative.

I suspect that the main issue here is the quality of feedback rather than whether the review is 'open' or 'closed'. Given human nature, I think that double blind, peer-reviewed feedback is best as it allows academics to be honest in their feedback without potential concerns of damaging relationships. This is particularly important when reviewing the work of academics who are in a reasonably small field. I have done a number of reviews and on occasions have reviewed the work of associate professors who did not submit work of a high quality. With the blind review process, concerns regarding anonymity or potential repercussions are diminished. So, in short, I think that the open process is a wonderful goal but with human nature the way it is, the blind review process may be here for a while yet.

Dear Helia Jacinto, Blind and open peer review processes has their own merits and demerits. I agree with your view point that the open peer review may serve the purpose more effectively.

Hello,

As an update to the conversation, you might be interested in today's editorial by Freddie deBoer, Can Double-Blind Peer Review Be Reformed?

Best wishes,

Barb

http://dish.andrewsullivan.com/2014/08/18/can-double-blind-peer-review-be-reformed/