It's worth reading the reviews their papers got. As they point out, the reviews are probably the most revealing evidence they obtained. All documentation is available here: https://drive.google.com/drive/folders/19tBy_fVlYIHTxxjuVMFxh4pqLHM_en18

and the reviews are collected in one of the folders.

Reading some of the reviews (there are a lot of them!) I am struck by the degree to which reviewers act as language police, ensuring that the language of the paper is closely aligned with an ideological position that is an unquestioned orthodoxy to which the journal and reviewer subscribe.

In many ways, this whole episode is a sociolinguistic triumph. Social groups have a language code, an argot, by which they can identify each other. By learning to talk like the members of the group, the authors (bless them!) were embraced into the fold. It didn't matter what they were saying, what mattered was that they were reading off the correct word cloud.

Health sciences have their own publication nightmares, with massive distortion of the evidence by commercial interests, so it's good to be able to take time off and have a hearty laugh at other disciplines. However, let's not over-react. The papers were published in journals where they could not possibly have made a difference to anything other than the journal's bank balance and the authors' chances of promotion.

I am still far more worried about the corruption of the health literature, where people can be, and are killed by greed.

The Sokal Squared hoax

by  Tyler Cowen October 5, 2018 at 12:20 am in Education, Science

Here is coverage from The Chronicle, the bottom line is that a number of humanities journals were trolled by phony submissions, and yes the journals accepted some absurd articles.

For varying perspectives, here is Henry, and here is Jonathan Rauch.  Here is Yascha Mounk.  Here is Bryan.

I would frame the matter somewhat differently, and perhaps more cynically.  Not every undergraduate major can have majors as smart and as rigorous as we find say in mathematics.  And yes I do mean some of the humanities majors.  In the resulting equilibrium, the rigor and smarts of associated faculty vary across fields as well.  The top people in quantum mechanics have passed through some pretty tough filters.  But again, we cannot usefully generalize those filters across all fields and majors to a country where such a high percentage of people attend college.  (Slow improvement can come from K-12 progress, of course, and we should fight for that.)  Some of the majors have to be easier than others, no names will be named.  By the way, don’t assume that basket-weaving is such an easy skill!

So simply calling for higher standards in the fields you object to begs the question.  Instead ask “what are those fields for?”  And “might I prefer a different kind of error process in those fields?”  And “Might I want those fields to be (partly) bad in a very different way?”  You probably have to compare bad against bad, not bad against “my personal sense of what clearly would be better.”

After such inquiries, you still will find that too much bogus work is being researched and published in journals.  The most rigorous fields in turn tend to have too much irrelevant or overspecialized work — is all of string theory or for that matter game theory so much to be envied?

Many of you will be inclined to call for fewer subsidies.  I won’t tackle that larger question right now, I’ll just note that any system-wide subsidies — especially egalitarian ones — also will boost the less rigorous fields and majors, and in some manner you need to be prepared to live with the not entirely rigorous consequences of that.

Overall I view bad pieces in the humanities as a potential profit opportunity, rather than something to just whine about.  You don’t like those troll-published pieces?  Get to work!

Addendum: You will note that the sociology journals were not fooled by the troll submissions.  By many outsiders sociology is a much-underrated field.

Best Regards Jason Deegan

Hi Osama,

Thank you for sharing - the link for the original article posted above by Osama is available for anybody who wants to read it here:

https://marginalrevolution.com/marginalrevolution/2018/10/sokal-squared-hoax.html

It's worth reading the reviews their papers got. As they point out, the reviews are probably the most revealing evidence they obtained. All documentation is available here: https://drive.google.com/drive/folders/19tBy_fVlYIHTxxjuVMFxh4pqLHM_en18

and the reviews are collected in one of the folders.

Reading some of the reviews (there are a lot of them!) I am struck by the degree to which reviewers act as language police, ensuring that the language of the paper is closely aligned with an ideological position that is an unquestioned orthodoxy to which the journal and reviewer subscribe.

In many ways, this whole episode is a sociolinguistic triumph. Social groups have a language code, an argot, by which they can identify each other. By learning to talk like the members of the group, the authors (bless them!) were embraced into the fold. It didn't matter what they were saying, what mattered was that they were reading off the correct word cloud.

Health sciences have their own publication nightmares, with massive distortion of the evidence by commercial interests, so it's good to be able to take time off and have a hearty laugh at other disciplines. However, let's not over-react. The papers were published in journals where they could not possibly have made a difference to anything other than the journal's bank balance and the authors' chances of promotion.

I am still far more worried about the corruption of the health literature, where people can be, and are killed by greed.

Hi Professor Conroy,

thank you very much for your contribution, it's incredibly helpful in shaping my own thoughts on the whole affair.

I wasn't aware of the concept of an Argot - it's incredibly interesting, and seems to stack up with the approach used by the hoaxers. Indeed it won't have much an immediate effect (although I am worried about the loss of trust in academia, that this feeds into - although this hoax I believe goes a way towards identifying the problem and hopefully a solution to that deficit).

Thank you again for your contribution.

Jason

I was remembering qualitative research done on science teaching in schools by the Nuffield foundation way back in the 1970s (yes, I'm that old!). They transcribed teacher-pupil interactions, and made an important discovery. The teacher will stop asking the pupil questions when the pupil presents an answer in which the term is in a syntactically correct sentence. So pupils are being taught to 'speak' science rather than to understand.

However, the reviewers' comments go beyond this, I think, into the Orwellian realm of policing the use of language. The authors' replies to reviewers are also interesting, in that they defend themselves by citing the work of others, with the implication that you can say it if you can cite someone else who said it. The fact that someone else expressed an opinion is not, heaven knows, evidence of its usefulness.

Dear Jason,

Thank you very much for your post and bringing to the forefront of this thread

the Sokal Squared Hoax. What follows is a short answer to your question.

1) The publication of a paper that was mainly mere verbiage and said almost nothing significant all all shows us that social and human sciences are really "soft sciences"

2) I think that if a paper similar to the focal paper but related to physics or biology, for example, would not be published in a serious journal of these fieds of knowledge. It migth even be the case that the Editor-in Chief did not send the paper to reviewers.

3) That the paper has been accepted to be published by all reviewers only reinforces my idea that social and human sciences still have to make many efforts that make them better sciencies.

4) The publication of the focal paper speaks in favor of its authors' smartness (intelligence?), and against the peer-blind review system, at least in the field of social and human sciences.

5) The publication of the focal paper also shows that there are many dispensable publications, namely in the field of humanities.

6) The publication of the focal paper also shows that reviewers can be available to publish what they do not really understand. The idea seems to be the following: "This should be a great and deep paper for I do not understand it!", even though authors are urge to write clear and clean papers,

7) The pubication of the focal paper also casts doubts on the so called peer-blind review system. As I see it, there is much inbreeding in this system. Once I wrote a paper arguing against, say, "sacred cows" in my field og knowledge and expertise. I experienced some difficulties with the publication of the paper. I had to make 4 revisions. I suspect that not all reviews were anonymous and blind because some reviewers -- I think -- were able to see that I was the paper's author.

8) Should the peer-blind review system be broken and replaced by a open (non-anonymous) review system? I am not sure about this. With all limitations, maybe the peer-blind review system is the most appropriate system of reviewing manuscripts. Open review can bring about revenge, which I hate,

I hope I has got your post and that this helps.

Best regards,

Orlando

Thank you very much everybody for your contributions, my concern is around how this highlights potential flaws in evaluation metrics used to understand the productivity and performance of academics.

If it's possible to get published by conforming to such rules, perhaps some academics are inclined to go towards these areas out of a desire for security (nothing at all wrong with this) but just that it casts a shadow perhaps in these fields on the tools of evaluation.