Henk,

You only have to read "Energy - The Subtle Concept" by Jennifer Coopersmith about the history of physics (from the perspective of energy) to find not one but several specific examples of critical and correct research that were refused publication. Known more widely it would have greatly accelerated the progress in physics. Instead it had to be reinvented (often several times), eventually by someone with an inside track to getting it published, and long after the truth of such research had been unequivocally established by other means.

Your view of journals and peer review being successful is one based on hindsight from a biased sample. You are taking the set of work that did get published to make your claim, when rather you should be analyzing the total set of research that could have been published.

Hence my counter-example above. Fortunately science history is not only contained in journals. You should be looking at false positive rates as well as false negatives. My concern is more about the winners that were NOT picked, and the impact this has on progress.

We are taught as scientists to analyse the data, the whole data, and nothing but the data, to make a clear statement of how we analysed the data, and to make that data and analysis available for others to replicate if they can or refute if they cannot.

And yet we then do the exact opposite of this method when the data-set happens to be the set of available research outcomes. We are selective about the data. We hide the analysis from view. We do not make the entire data-set or the analysis available for others to replicate or refute.

Data is data. The method is the method. It has stood us in good stead. Why do we have this blind spot in us when it comes to using it as the process by which material is published?

This is a tricky question. If it were a scientific process the participating referees would come to the same (or at least to a similar) response. That's mostly not the case, as has been stated in many posts in the thread linked below. Respondents submitting their review are driven by many circumstances: schools of thought, knowledge of the field, mood, daytime, awareness being fair and/or polite.

On the other hand, it's obvious that peer reviewing is part of the scientific process. You have a result (the publication, after hard work and much effort by the authors) that is being reviewed (discussed in some way), similar to the discussion of results in scientific papers.

So, to answer your question, I think peer reviewing is a sub-process of scientific endeavour, and to include a remark on your additional comments, false experts are not only publishing but also reviewing.

https://www.researchgate.net/post/What_are_your_experiences_relating_the_quality_of_peer_reviews_for_your_papers

Indeed, reviewing is voluntary work - and now, with the pressure on academics to publish, and no one is appreciating the work of reviewing - I am afraid that this voluntary work will diminish and that is the time when a scientific system might be put in place! another impact factor and another ranking !!!!! daaaaaaaaaaaaa

Theodora,

Yes it seems most journals quality fall on the dedication of a few...and this is a variable thing depending on personal circumstances and this dedication is on the decline, since it does not seem to be its own reward!

I cannot help but think publishing the reviewers coments alongside the research both good and bad might have merit. This is how movie ratings work....and it means if you trust a reviewer, a bad review will downgrade the score. It also keeps an unbiased historical record of science , and permits the performance of the reviewers themselves to be scrutinised. Humans seem to be their best when someone can look over their shoulder at what they do!

In my opinion peer review is a scientific process. The problem is in the people who work in the process. Yes, they are volunteers... but some reviews should not be accepted by the editors. I seen reviews no longer than 2 rows that have nothing of scientific.

It's not necessarily a 'scientific process' - but there is supposed to be a methodological and rigorous process; although that will vary from journal-to-journal and reviewer-to-reviewer. Good response from Michael as usual. Andrew - it's safe to say that you question is a little 'leading' and you haven't always had a good experience from reviewers. As soon as you say '...to gain popularity over plausible but idle speculation by false experts.' - the tone of the question has been set.

Dean,

Actually the papers I have wanted to publish I have had no problems with, So it was not leading at all from a personal agenda. I have had occasional problems as a reviewer though, both with well written papers that contribute nothing, but have ticked all the right boxes for publication, and with those poorly written but with the germ of a great idea, knowing you are going to go through a lot of unpaid grief to help get it published.

I do happen to be quite concerned about the state of the industry and would like to think we could do something positive about it. I accept the peer review process has by and large stood us in good stead, but I remain unconvinced in the modern era that we cannot do a lot better, and this is definitely what you would interpret as 'leading'

I am a great fan of transparency, and truly believe that publishing reviewers reports would go a long way to addressing the system by allowing readers to judge the merits of the review alongside the actual research. Also, to create an unbiased account of the history of science, we should publish the good and the bad - with the bad being flagged as such by the review documents and discounted accordingly, also as examples of what NOT to do. we seem to have no problem publishing bad movies that flop, and nobody takes undue notice of them - same with research. And who knows. It might turn out occasionally the bad research actually had a valid point buried somewhere in it which creates that spark of "a ha" when someone else stumbles across that single sentence.

And it is not as though false experts seem to need the bad research to make up their thoughts - they just make it up as they go along anyway. The problem with false experts seems to be caused rather by the hiding away the good research so that there is not an easy way to counter the false expert hype.

Which brings me to the subject of this sickness we call paid firewall nonsense on the internet. I used to be able to walk into any University in Australia as a member of the public and pick any journal off the shelf and read it. Now the same stuff I can't - and I cant get a refund if the paper abstract does not deliver on the promise. in other areas we have consumer protection laws against such nonsense. Productivity gains have made the process more affordable, not less, so what gives! - something I use to be able to do, which has got cheaper, I can no longer do? Surely some ideology as well as economic greed is at work here.

Andrew - that makes it clearer - thank you. Overall, I'm supportive of your comments - and I particularly like the analogy of relating the process to the film industry. It helps to strike the problem home. Henk - I agree that reviewing is a subjective process - just like marking student assignments. Not sure what the answer is. There does seem, at least on RG, a growing consensus for some form of reviewer 'peer review' - but the flip-side to that (as reviewing is generally voluntary) is the likelihood of a diminishing pool of reviewers. Going back to my earlier point about the subjectivity of marking studnet assignments - most universities have a generic, structured marking framework and policy that helps to reduce that subjectivity. It would be nice to think that good journals could develop and incorporate similar policy and structure but, again, how easy would that be to implement and enforce? What about a rating/grading system or checklist for published articles within a journal - rated by reviewers, editors and the editorial board for a 'final score'. Even within the best jorunals, we know that all articles are not 'created equally'. That could reduce the type of frustration that Andrew describes of the article not backing up the abstract.

Henk,

You only have to read "Energy - The Subtle Concept" by Jennifer Coopersmith about the history of physics (from the perspective of energy) to find not one but several specific examples of critical and correct research that were refused publication. Known more widely it would have greatly accelerated the progress in physics. Instead it had to be reinvented (often several times), eventually by someone with an inside track to getting it published, and long after the truth of such research had been unequivocally established by other means.

Your view of journals and peer review being successful is one based on hindsight from a biased sample. You are taking the set of work that did get published to make your claim, when rather you should be analyzing the total set of research that could have been published.

Hence my counter-example above. Fortunately science history is not only contained in journals. You should be looking at false positive rates as well as false negatives. My concern is more about the winners that were NOT picked, and the impact this has on progress.

We are taught as scientists to analyse the data, the whole data, and nothing but the data, to make a clear statement of how we analysed the data, and to make that data and analysis available for others to replicate if they can or refute if they cannot.

And yet we then do the exact opposite of this method when the data-set happens to be the set of available research outcomes. We are selective about the data. We hide the analysis from view. We do not make the entire data-set or the analysis available for others to replicate or refute.

Data is data. The method is the method. It has stood us in good stead. Why do we have this blind spot in us when it comes to using it as the process by which material is published?

Dean,

" most universities have a generic, structured marking framework and policy"

Very true, but this in itself has been largely subverted by the fundamental changes in the nature of our Universities. They have changed from:

institution of excellence - teacher - student

to:

service provider - client - employee.

Note well the swapping of the order of student and teacher.

Any attempt by the teacher to improve standard (or heaven forbid suggest a student does not meet the conditions of "the framework provided") is immediately countered by clients rallying to their service providers and the framework being downgraded so the student does in fact meet "the framework provided"

Is it any wonder morale is so low among teachers in universities?

Hello all,

Interesting discussion - well Andrew - this is a very frank statement:

"Is it any wonder moral is so low among teachers in universities?"

Regards

Theodora,

I have corrected moral to morale LOL, as I meant "esprit de corps" No moral judgments here folks, go on home! Just one less character on the keyboard when I type too fast.

The situation is changing: Morale is improving because teachers are just thankful to have ANY job, and because those teachers who were seeing a fundamental problem with students negotiating with universities to set teaching standards for teachers have given up and moved on. The universities now employ teachers more compliant to this view, and therefore more happy in their job!

Also my perspective is from 12 years of post-graduate research teaching while this transition was happening in the 1990's, rather than from the undergrad course work teaching today in 2013. No it was not due to bitterness as "grumpy ex-teacher" because I have not taught anyone! I was a research engineer, observing the thesis supervisors and students alike and what was going on around me during my day job.

I do not mean it as a complete generalization - there is a Bell curve and I would suggest your Curtin and ANU are at the more positive end of it.

Henk,

There is no doubt that peer review and a stamp of quality in a journal publishing process have been valuable contributions to progress in science at certain times and places. I agree with you on this point. It made for as good and as scientific a process as the good-will of the people like yourself encouraged, which is why some journals can rightly be called 'great" publishing works.

My concern is that time has moved on, and we can no longer rely on that good will and personal commitment to do the right thing in journal publishing as we have in the past. The whole media and communications industry is in turmoil at the moment in the face of this great convergent gush of current and historical information that we see on the modern internet. It is obliterating whole industries almost overnight with new (not by definition necessarily better!) business models. It is my view that science publishers in this recent rush have got it almost completely wrong, and are taking the more scrupulous publishers down the rabbit hole with them.

We could just sit back and let some form of massive group consensus try to sort this out. But culture and context is a very fragile thing, and bad culture and context is very hard to get rid of once established. This thread is really about trying to set some ground rules for "the new way of science publishing" and to me, looking to the scientific method is as good a way as any to discover what these rules might be. In science publishing we desperately need something right now that is both "right" and "popular" at the same time.

Hmm, I am just reasoning.

First question is, how strictly you want the research assign to the scientist: On one side you could reason about a freaky approach: open a network like wiki, where everybody can post his/her findings and the discussion replaces (hopefully) the review process, and even helps to evolve the research and/or the idea/hypothesis behind it. The flip side would be that at the end, it could become a needless mess or a good new approach, where many brains were involved. Yet, in the latter case it would be impossible to decide who added how much to the final goal. If you want to keep it as it is, step-wise pushing knowledge further by publishing work–study by study–published in journals, where every step is closely anchored to its author(s). You can track the authors for their contribution, as well as for the quality and/or quantity of the scientific input they provided. Yet to do so, in my eyes, you need the journals and you need some peer-review system to give the journal's editors an external feed-back or input.

Second, why not recognise the peer-reviewers work, but thereby also take them into responsibility. While many researchers run for impact factor 'at all cost', why shouldn't there be a quality measurement for peer-review? Let us think about a single formula. For every peer-review done, the editor should not only give the reviewer a nice feedback ('Many thanks, blah blah, blah', and a citation in the journal's last annual issue), but give him a rating (e.g. from zero for "the peer's review contributed nothing to help the editor in his/her decision", to 5 for "this review was done proper, and maybe even gave input to change the way of the editors first impression of the manuscript under review"). Can be harsh, but would be a clean feed-back for a peer-reviewer at all. A feed-back I often miss, when I do so. Now go one step further, and multiply this number with the journal's reputation and add this score over a reviewers/scientists career. Maybe this would help to sort out 'slackly' review quickly, and thereby would force peer-reviewers to consider their work twice. Yet, if a peer-reviewer staidly delivers good work, it also would give some kind of merits. At the end you even can call it part of the scientific process (again).

Bruno,

I like your "messy wiki" approach far more - the goal is to get to the "true idea" clearly expressed, rather than who said it first.

I do agree that there is an important "categorization of research" role played by journals to keep the field of study focused enough to make progress 'in the detail"

But there is a role for generalists, specialists, and cross-disciplinary fertilization, the first and last not being served well by the current system.

I would add to your "messy wiki" approach - we are sorely in need of a formal categorization of meta-tags so we all know for sure when we refer to the the same thing, and when we refer to something different!. I guess #tags in social media are a sort of group consensus way of agreeing on the metatag for a given 'concept"

I would love to see RG expanded with the right tools so that real research could be undertaken on it, as compared to just talking about it.