As a researcher, every so often one encounters publications that are not of the high quality that one might expect from a 'good' publisher (such as the IEEE). Sometimes it is simply a low quality workshop or conference, where peer review is simply not that high a bar compared to high profile conferences or high quality journals. We all know there is a difference in quality between each workshop, conference or journal. This is something we can deal with- lower quality work is not necessarily wrong, but it is just often less innovative, or it is on-going research.
However, very rarely, one encounters publications that simply contain many errors. In this case, I'm referring to errors in the content, rather than spelling and grammar (because spelling and grammar errors are virtually everywhere). Sometimes 'errors' are simply assumptions you do not agree with (which is acceptable most of the time), but in other cases, 'errors' may refer to falsifiable facts (for example, claiming to be able to protect integrity by using HMAC-MD4 -- MD4 has been known to be broken for many years, and this algorithm isn't part of major communication security standards like TLS).
What do you do in such a situation? Do you ignore the paper, or do you cite it and reason why its solution is bad? However, citing increases its citation count, thereby increasing the legitimacy of the paper - a confusing dillema, for me at least.
Secondly, do you think the world would benefit from a place where this type of (motivated!) criticism can be published? Something like arxiv, where authors can publish technical reports explaining the issues, with the option for the original authors to respond to the reports as desired? Or would this lead to a strongly negative spiral that only serves for authors to attack each other? Finally, could ResearchGate fullfill this positiion, using its significant userbase as a kick-start?
I Think this will turn out into a Platform to Attack authors, but will be beneficial for all. Good thinking altogether.
Thank you for asking that question! I've been thinking about this as well for some time, but I've not come to any conclusions yet.
I do think that posting article-size responses on arXiv.org without going through peer review first is dangerous. Every once in a while, anybody has a really, really bad idea. It's good for everybody involved if such things are caught before being circulated!
I agree with you that "boring" papers are not necessarily bad. I prefer a solid work with boring outcome over an "innovative" piece full of errors!
I think your suggestion of a culture of commenting here on ResearchGate, like in blogs, would be nice for academic papers. Others could ask questions or voice concern easily and more "low-profile", and a reader can get a better image of what the community thinks of a work. I always enjoy reading the comments on a controversial journalistic online article. At the same time, the criticism would not boost the impact of the paper like traditional responses.
Related to this, I think that both authors and journals have an interest to post "troll" papers, i.e. papers that everybody will have an opinion on and that contain subtle errors that will create a lively discussion between believers and critics. The infamous "arsenic bacteria" paper in Science < http://www.sciencemag.org/content/332/6034/1163.short > is in my opinion a perfect example, or the 10^8 improvement of ionic conductivity < http://www.sciencemag.org/content/321/5889/676.short > (also Science...). I do not take Nature or Science seriously anymore because they publish such stuff with obvious errors that should never have been printed. In my opinion, it is clearly either trolling, or utter incompetence of the authors, editors and reviewers. I'd like to have a downvote button for such nonsense!
Great question Rens,
But i would think that any proposed solution will do far more harm than good.
Pretty soon you'll have a situation where folks with an agenda will attack perfectly OK publications with specious arguments (I'm sure you've seen it before - so you'll have to police the police, ad infinitum). At worst, this will put a damper on academic freedom as people will become leery of pushing the envelope, lest they be attacked by the hordes with an agenda.
The system is extremely good at annihilating, over time, bad contributions. Remember a few years ago the Bogdanov affair ? That was extreme, but I've also read my share of patently wrong contributions - even books ! Over time, these have invariably sunk without a trace, regardless of how they may have created a buzz when they first appeared. In the meantime, they played a positive role - exercised readers' critical judgment, made for a good laugh, and so on.
Please do not build in one more layer of censorship - please read David Kaiser's 'How the Hippies Saved Physics' for a book-length exposé of why upholding full freedom is so crucial.
If the price of freedom is that once in a while an arrantly wrong contribution makes it through the meshes - so be it.
@Jens
First, I would like to thank you for you observation, "To my knowledge, most journals allow "comments" or "letters to the editor" to be submitted as a peer-reviewed means of commenting specific articles and formally critiquing them. Since you mention IEEE I suppose you work in some type of technical/engineering field, and I am unsure whether this tradition exists in said field - but it is somewhat common in e.g. the health sciences, and I would be surprised if it isn't possible in most other fields as well." It was the first thought that came to my mind when going through long explanation of the post. Though, I am not an engineer or the scientists but a regular user of IEEEXplore which provides articles per search keywords. However, If one goes through the journal in which an article is published, he can encounter commentaries etc. Same is true about, Scitation or Digital Library of American engineers in different fields, if one goes through the entire journal, one may find commentary on the articles published in the previous issues. It is standard practice of professional journals.
i think the problems are deeper - many journals, including high profile ones, have their agenda and sometimes 'inner circles'. the key problem in social science is that you have 'guardians' of knowledge and creativity, and when they are not true guardians, then the situation is hopeless, I think. I had some strange experience, for instance, with the Harvard Negotiation Journal, submitted a paper and never received an assessment. I think they simply waited for me to un-submit. Causes and motives? Also, creativity is a problem - I think that academy stifles it (as a rule to which there are exceptions). The higher education sectors are a kind of compromise between governmental needs and the state of knowledge at the moment.
First off I wish to thank everyone for their comments. I wanted to respond to some of them earlier, but I figured I'd put everything together in one post.
@Dieter/everyone:
I mainly suggested researchgate because, at least in the past, it was possible to directly link questions to papers. However, I think noone really used this feature (and when I tried to use it, I couldn't find it anymore). It was the closest thing I knew to what I intended, hence.
@Jens:
Indeed, your quote emphasises exactly what my point was, asking this question.
With respect to reporting (towards the IEEE): In the IEEE at least, there is some support for reporting of serious misconduct (such as plagarism); however, this requires the submitter to write the editor of the conference or journal. Although I haven't finished my search, I believe it isn't easy to really publish a formal discussion, as opposed to an accusation of misconduct. I'm fairly sure some of these notes I have are definitely me misunderstanding the work, as opposed to actual errors. But I'd still like to get them out somewhere, for two reasons: (1) it will help the authors understand how their paper is read, and to improve their writing skills; (2) it would allow for critical feedback.
@Chris:
To be clear, I'm not proposing a layer of censorship, but rather the opposite - publishing ones questions and comments as a seperate type of work. Of course I can always do this on arxiv, or on my university's webserver, or on my own, but the idea was to do it in a more "supported" way.
The problem with letting things fade away is that in a fast-moving field like computer science, you need to be right on top of each new publication in you field, and at that time the distinction has not yet appeared. This is a real issue for new researchers (as well as students that need to write theses and student papers), as they can often not distinguish by a detailed reading of the work. This is also why directly accessible feedback would be important - ideally, the publisher would simply attach relevant comments (as seen on the links to sciencemag.com that Dieter provided).
@Matthias:
Thanks! I didn't know about pubpeer.com yet. I'll be sure to check it out as soon as I have time.
For me, this would depend on the nature of the error:
-Oversight or misunderstanding of the authors: How about just contacting the original authors and explaining the issue. You can then still reach your goals (1) and (2). No need to shame the authors publicly. The authors can then possibly make updated version available.
-If the paper is funded on that error, and generates a lot of interest and gets cited a lot, you can then publish a detailed report (after possibly removing misunderstandings etc) alone or together with the original authors. This will allow the authors to publicly point out their position, without starting a war. If they decline (and disagree), publish alone. If you are right, and they have nothing to add, publish alone.
- If proving the error itself has scientific contribution (i.e. requires novel experiments, etc), then go ahead an publish somewhere. You can notify authors in advance, but you probably made valueable contributions in your work
Another aspect of this problem is the difficulty of distinguishing between good and bad research by scientists working in interdisciplinary fields. If you are working in between disciplines, you often do not have access to the discipline-centric discussion of new work and informal evaluation of new papers that you have in single discipline topics. These discussions often lead to widespread opinions on the merits of a piece of work without anything ever being written about the arguments for or against.I would prefer to have some commentary in the public arena of a paper's merits so that scientists outside of the discipline can have insight into what is thought of an analysis, approach or method.
I think Molly Brown brings up a very important point of distinguishing between "merit" and the more technical discussion of papers I was originally aiming for.
I think a discussion like this one should never focus on the merits of the work, but rather on making explicit the underlying assumptions and methodology that is used to come to a certain conclusion. The merit of a work should be decided by the funding agency, rather than by the publisher: after all, the funding agency is the one interested in the research being done. Note also that the merits of a work are extremely subjective - just today I had a discussion with collegues from the vehicular networking community about the merits of privacy research within these networks. Many argue that privacy is lost in any case, due to a multitude of other tracking systems, while others (incuding myself) argue that privacy is essential to the adoption of such networks. However, this discussion is completely seperate from underlying technical assumptions contained within such privacy research - for example, the average transmission range, the amount of vehicles that can be expected to be in one place at any time, the amount of infrastructure support that is available, and so on. Even these are subjective, but they are at least quantifiable and they can be input parameters that further researchers can use to decide which privacy mechanism should actually be implemented.
@Vladimir:
I will probably be posting to this website in the future. However, I think I speak for many researchers when I say that time is the major problem. It costs at least a few hours to collect and put together a solid and detailed argument against a bad paper. Sometimes these bad papers look like they took less time to write than it took to refute them.
Apart from that, many researchers are doing research as a job, not as personal entertainment. While some of us are willing to dedicate some personal time to improve science (e.g. through discussions on researchgate), our time is not infinite, and many of us also need to write a thesis. To put it bluntly: time is better spent contributing new research than it is crushing bad research.
I agree with you that this is not at all true, but the sad truth is that from a career point of view, criticism on others works is much more likely to have a negative impact than a positive one. Why do you think we have double-blind reviews? Yes, we should have all these changes listed in the article. In an ideal world, we would start them today. But in an ideal world, we do not need them- and this world is not an ideal world. It remains the case that businesses and other grant issuers are interested fundementally in money and (maybe) results, not in accuracy or quality. The same seems to apply to publishers. So what do we do about this? If noone is willing to pay for it, can we rely on researchers like you and me, willing to put in their free time? I'm pretty sure this helps -just look at the open source community- but it is not a complete solution to the problem.
On the positive side, I do every so often see a paper in the IACR archive (authoritative archive for cryptography tech reports and pre-prints) that debunks security proofs of other works. This - 2-3 page documents in a paper-like format- helps and is what I was aiming at with the question. Of course, the issue of time remains. There are hundreds of workshops publishing relatively low quality research - many of which were never intended to be seen as authoritative, but when we tell students/new researchers/the world to look at actual publications, they will find these works (too). Not to mention potentially fraudulent research in journals, which I believe is another issue entirely - malicious intent is harder to defeat than sloppy reviewing!
With respect to time: I can easily write a discussion here inbetween discussions I'm currently having, but writing a structured argument takes much more time and concentration. I can't just write "hey, this paper is really vague on this aspect and that assumption is complete nonsense". I need a structured argument with references and details, showing exactly why the vagueness of these aspects are problematic, what the potential impact on the results is and how such aspects could be clarified. Similarly, I need a founded argument why certain assumptions will/may not hold. This is how you write reviews- so I feel it should also be a minimum standard for such "complaints". I must be completely certain and I cannot allow an error to slip into such a document. It is like writing papers - you don't want to ever make mistakes in these documents.
Secondly, a very important issue is also the impact on ones career. I think it is safe to say that publishing many complaints about papers is not a good way to advance ones' career. This isn't as much an issue at the website in question - where some pseudonymization occurs - but one should always keep in mind that a registered account means at least someone has the capability to do serious damage to your career. Perhaps that is a paranoid view, but as a computer security research, I *should* be paranoid, it is a bit part of the job ;).
Last, it is of course an issue surrounding my free time. I don't want to spend my time bashing a paper, because it makes me feel like I'm only making negative contributions to the world (irrespective of whether that actually makes the world better), which is a bit of a psychological barrier. Maybe that's also making excuses for not working- but hey, it is my free time, not my work time. If it were possible to do this as a real contribution to science, e.g. as part of my job, then I would definitely be more inclined to actually do it.
(edit:) Also, it is required for this website to have at least one publication - my first publication is accepted but not yet published, so I'm not capable of actually contributing at this point in time. Publishing will at least not occur until after the conference (likely only months after that), so up until then I have an additional barrier.
I concur with all these responses. The easiest means to address this issue (from my experience and that of others) is to simply write a 'letter to the editor' or a 'correspondence' piece, usually 500 words or less (thus not too time consuming, you can get a small publication from the effort, and you can promote your own work that you believe is of higher quality, etc.). Keep in mind, though, that you need to 'be on the ball' with such publications, meaning that you must submit a 'letter to the editor' or correspondence piece very shortly after the article in question is published or made available ahead of print. Email and GoogleScholar alerts are helpful in identifying these articles in a timely fashion.
With respect to privacy- it is much more interesting to use text analysis to break pseudonymity than it is to track IPs. But I think this may be moving our discussion too far off-topic. As for the two-line comment - I don't believe I can make a substantial contribution to either discussion or critical analysis of any paper with short comments. This works in real life discussions (because you can't explain much more without a prepared talk), but it is far more difficult to do this effectively through text. A good discussion requires commenters to go into some detail, not just writing quick comments.
If you have spend time to replicate previously published experiments and found conflicting results later then the first months of the publications of the previously study you may need to write a full paper about it.
It is properly not going to be an exciting paper by itself so it would be good if you come op with a experimental solution to the problem and publish it too, so the negative only becomes ½ of your paper.
I thought this might add to the discussion:
http://pipeline.corante.com/archives/2013/02/21/an_incentive_for_hype.php
http://bjoern.brembs.net/news.php?item.899.3
Basically, the pressure to publish in high-ranking journals drives researchers to overstate their results.
@Vladimir: because this is not a scientific discussion. This is a bunch of people sharing their personal insights on this topic. Scientific publication should be (much!) more rigorous than that; arguments that make claims against already published works should be even more rigorous.
@Dieter: thanks! I think both of these articles very well illustrate issues I initially had writing my master thesis. However, I would like to add to that: it is the responsibility of the citing author(s) to at least have read the paper, and did some basic sanity checks on it, if it is important to their own research rather than some survey that explains the problem and existing solutions. Perhaps it would be a good thing to produce a little more awareness of what you do when you cite a paper- you are extending upon that work, so you better damn well make sure you're convinced it is right!
@Rens: But for that I need to know a lot about the subject of the paper already. In the areas that I am competent, I notice flaws quickly. When I look at papers from outside my core core area, I lack the background to check it. That leaves me with a bad feeling in my stomach, because I know that I cannot trust anything I am reading. From my core area I know what kind of stupid mistakes people make...
- Badges
- Science topic
- Similar topics
- Publishing