I came across an article which claimed that most research findings are false. I had difficulty understanding the article completely. However I will quote some lines from the abstract of that article. These lines say that
"... a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias."
What are your thoughts?
p.s I have attached the full article downloaded from http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124 with this question as an attachment.
Dear Munir, While I do agree that there is some truth in the tale, the title is alarmist, and any decent referee of the paper would not
have allowed the title to be accepted. The chosen title would imply that more than 80% of findings of all medical and non-medical research is not true, which is patently not true.
The title is not proven to be true in the essay!
A better title might have been “Reasons why some research findings are not true”.
Also the essay focusses only on the concept of “false positives”, but this did unfortunately not make its way into the title.
The essay should be read as a piece of opinion, or as an essay, not as a research paper. (Read the small print).
Remember too that the essay reports on a simulation, not on a meta-study of research papers or on case studies.
Strangely the paper has no conclusion section; it ends abruptly with a discussion.
Yes, we do need to watch out for inadequate or biassed samples etc.
However not all research uses samples or is biassed.
For further clues, see http://en.wikipedia.org/wiki/John_P._A._Ioannidis
The statement that " all published research findings are false", is a bit too blatant ! It's true to some extent that small sample size etc may not prove facts and hypotheses but these are few examples! Generalised statements made may create controversies !
I agree with Ian and Jaya. I also voice my concern that the scientific field needs more stringent rules on allowing papers into their journals with regards to:
1) Reproducibility - I think that one of the reasons this is so is because of maximum number of pages. A balance should be made between completeness and number of pages
2) Lack of solid statistical testing for significance
3) Lack of poor modeling- sometimes statistical tests are not enough and good models to explain the phenomena are needed to support the findings.
4) Poor mathematical proofs- either because the author thinks its obvious and it is not (therefore you cannot deduce what he did) or just plain wrong constructions.
Each of these should be tested from a sample of journals (such as A Quantitative Study of Experimental Neural Network Learning Algorithm Evaluation Practices Lutz Prechelt. It is a dated study, but I did a survey on 27 articles on neural networks and intrusion detection systems a year ago and found that this trend was still very much the same).
While I think the article is right , I think it made the same mistake it is criticizing :-)
It might be possible that researchers spend too much time and expect a lot from there project work but unfortunately if they not get positive and significant result, they use some loopholes to cover the results. But most of the time results are positive.
The key issue is reproduceability. If the results can be reproduced a sufficient number of times by independant research then we might start to believe there is something in it. A single study with no checking for reproduceability is just an interesting suggestion. That almost all the published literature has not been checked and so not proven to be true is alarming but not alarmist.
There is also the problem of false correlations. If the original hypothesis is not supported by the data then a trawl for correlations to get something to publish is often used. Random matrix theory tells us to expect false correlations in any data set and the larger and supposedly more reliable the data set the more false correlations are to be expected. Even more worrying is the random chance that the verification of the hypothesis may be one of these false correlations. This is without all the biases and self-decption that human beings routinely perform without any awareness of what they are doing.
Statements like, "But most of the time results are positive." scares me as much as those famous 6 last words, "I know what I am doing." When I was a kid and we used to play with improvised explosives we always had a safety overseer and asked someone else to check every move before we made it. That is how I got past my 13th birthday. If we all took the same attitude we might be able to feel more secure about our knowledge.
The article proves one thing beyond doubt; there is at least one article out there that is not only utterly rubbish, but also false!
- Badges
- Science topic
- Similar topics
- Computer Science
- Research Topics