See also: https://stats.stackexchange.com/q/398646/166526

This is not new. In 2005, PloS Medicine published "Why most published research findings are false" by John Ioannidis. The issue with p-values was not even new then.

In a recent thesis defense a faculty person in the audience wanted the student to discuss the outcome from a graph that clearly showed that the outcome of treatment A was larger than the other three treatments and in his words "ignoring the statistics." Yet they were uninterested in the next graph that showed a different effect (also non-significant) because that outcome was not in line with their belief. If we can just pick and choose, then science is meaningless. Given this skill level, are you sure that a different approach will yield a more desireable product?

Maybe we could report all findings, but allow the 0.05 significance tradition to help focus the discussion. Then work on the social issues like "it was published and peer reviewed and therefore true."

We need to report/publish non-significant outcomes, but only for exemplary research. I don't care about non-significant outcomes from two replicates and an inappropriate model.

Statistical Inference in the 21st Century: A World Beyond p < 0.05

https://www.tandfonline.com/toc/utas20/73/sup1?nav=tocList&fbclid=IwAR3Bqo-hUAvrvvpHRfDUezn102TTn0DfOzAZmoG48cBaciIU2aOVJz_EwpM

For your mention of "51% of reviewed papers #erroneously stated "no effect"", I think you are missing the point being made. Those papers are presumably not making the correct distinction between "no effect" and "no conclusive evidence of any effect".

David A. Jones thanks. But what is the difference between the two? A larger p-value means that the new treatment is not having any effect = "no effect" and "no conclusive evidence of any effect". To me both means that the hypothesized effect is not apparent, meaning that we do not have convincing evidence to reject the null hypothesis.

I thought the concern pointed out in the comment was issue of consistency, reproducibility and repeatability of conclusions from different researches of the same or comparable setups. For that the board of the Association of American Statisticians (ASA), issued a statement in 2016. They pointed out that the inconsistency of conclusions are NOT due to the p-value or hypothesis testing process models but the research design and data analysis practices. ASA believes that statistics is misunderstood and misused as it is done by people who do not have adequate training in research design and data analysis. ASA listed 6 points to help statistical practitioners and writers, which talks about statistical models and their assumptions, sample size and analysis powers, research design, data quality, reporting the entire design and complete data for publications and avoiding reporting of only the more appealing part, what they also termed as "cherry picking, significance chasing, etc".

I expect that ASA will have a say soon on the comment in nature.

http://web9.uits.uconn.edu/lundquis/ASA%20statement%20on%20p%20values.pdf

Tadesse Fikre Teferra . The difference is substantial. Recall that such a statement is meant to be a summary conclusion, understandable by someone who hasn't followed all the details of the analysis. The difference is that between:

(a) I am sure that there is no effect;

(b) I am not sure if there is any effect, but if it is there we couldn't find it.

This is partly a question of phraseology, similar to the philosophical difference between a null hypothesis being "accepted" or "not rejected", which I believe has been a question as to which is correct. I thinking current teaching practice is to go with "not rejected".

The points of concern outlined in the original question are all things that need to be got right, in addition to questions of accounting for multiple testing, etc., but the point I think is being raised "in the supplemental data" applies regardless of any details of the statistical analysis.

"not rejected" is the correct answer. Maybe this has been argued between non-statisticians, but there is not any issue within the statistical community (which is the one that matters).

There is "no effect" is usually interpreted as a claim that the null-hypothesis is true. Using the phrase "we could not find it" leaves open the possibility that increasing sample size or otherwise changing the experimental design would make clear the expected effect.

If you conclude the null-hypothesis, then you are concluding what you assumed in conducting the test. The p-value is the probability of getting a value as large or larger than the observed value given (=assuming) that the null hypothesis is true.

Timothy A Ebert You are perhaps too young to know/remember that, until relatively recently (70's, 80's) , mainstream statistics text-books written by statisticians were perfectly happy to use "accept the null hypothesis", with little or no expansion/explanation in the direction of current usage. There must have been some point in time when the "accept the null hypothesis" usage was found to be too misleading to outsiders.

Nonetheless, it is easy to find up-to-date teaching material that is still happy to use "accept" .... for example:

https://internal.ncl.ac.uk/ask/numeracy-maths-statistics/psychology/introduction-to-hypothesis-testing.html

http://www.indiana.edu/~educy520/sec5982/week_12/hypothesis_test_summary011109.pdf

My first introduction to statistics was about 1983. I learned that I can accept the null hypothesis, and all I need are 4 replicates. We still accept the null hypothesis in things like testing for normality: if you cannot reject the null then that proves that you are dealing with a normally distributed (variable, residual). And we can prove a normal distribution with four replicates. There was no discussion about power, or being able to reproduce the results. No discussion about missing data. None of the students even asked, and I was no exception.

Then I made the mistake of taking more statistics classes, last one about 1993. I still use a bit of the material from the first class, but mostly it was all modified by later classes.

Now I have to work with people where the paradigm is 4 replicates. They are very certain that what they were taught by a faculty member in the statistics department is true. On good days they plan four replicates and afterwards have a great idea of adding more variables. Afterwards I see the data and find that a few treatments have only 1 or two replicates because there were problems. They know how to design experiments, so all I get is the data at the end. I also know that this experience/complaint is common.

Just because it was taught, does not make it correct even if taught by a senior level statistician with a Ph.D. from a fancy institution. Just because it was printed in a textbook, or a peer reviewed journal is no guarantee that it is correct. There are plenty of mistakes in the literature. Yet somehow we still seem to make progress. Not every medical study is true, but I would prefer modern medicine to what was done 300 years ago.

I stand by the statement that a failure to reject is not proof of the null hypothesis, and therefore "not reject" should be used rather than "accept".

Timothy A Ebert We might agree, but you said "there is not any issue within the statistical community", whereas there has been a lot of discussion around this issue in the statistical community, in the past at least, leading up to the note in Nature Communications, and earlier stuff.

One could point to the series of YouTube videos from the Royal Statistical Society, published in 2018:

https://youtu.be/VRF-UwrepAs

https://youtu.be/R7dQo9bW1DM

https://youtu.be/gtQlvR-RUPg

https://youtu.be/txLj_P9UlCQ

And of course, the publications noted above by Keith Godfrey

Thank you Keith Godfrey

Here is the link: https://amstat.tandfonline.com/doi/abs/10.1080/00031305.2019.1566091#.XKvGcujYp9A

On the idea of "Accepting the null hypothesis", I remember my stat professor from a couple of years back, emphasizing that we DO NOT accept the null hypothesis. She was saying that in a hypothesis testing process we either:

• Reject the null hypothesis, which can also be termed as "Accepting the Alternative hypothesis"

OR

• We "Fail to reject the null hypothesis" which IS NOT equivalent to "Accepting the null hypothesis".

The idea is that the null hypothesis is what is assumed to be the general/existing trend or behavior of the population before the application of the treatment. The alternative hypothesis is that the treatment results in a new set of behavior(s) and that is all we want to show in researches.

David, at least in the first youtube example (Deborah Mayo) I was unable to find a discussion saying that some statisticians argue that a failure to reject the null hypothesis was proof that the null hypothesis is true. I don't think this is what was being said at time 17:34 " Significance testers object to highly significant results being interpreted as no evidence against the null -- or even evidence for it." However, maybe I misinterpreted it?

.

deciding what to state about a hypothesis when it is not rejected is an inductive operation

it is easier to discuss this under the Neyman-Pearson framework, and, even in this framework, things have to be carefully handled

i found the paper below very useful in this respect

.

Severe Testing as a Basic Concept in a Neyman–Pearson Philosophy of Induction

Deborah G. Mayo and Aris Spanos

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=7CAA49CCD3F03DA7759D6AC963148B29?doi=10.1.1.130.8131&rep=rep1&type=pdf

.

section 7 quickly (too quickly ...) discusses the case of Fisherian framework (null hypothesis versus "rest of the world" testing)

.

Sad that what is (or at least should be) taught in every 100 level statistics course has become the fodder for a popular debate.

Either the null hypothesis is rejected, and the size of the effect (based on confidence intervals, or credible intervals or other uncertainty bounds) is reported, or the authors clearly state unabashedly that the data and experimental design were inadequate to draw reliable conclusions about the effect under consideration. And a point estimate and confidence interval should also be reported in this case as well. Were the estimated parameters far apart biologically, but imprecisely estimated, or were the parameter estimates close and differences precisely estimated. This is what we really want to understand, so report the intervals and interpret them, and no harm in identifying the level of significance of the test....The primary problem is failure to communicate the scientific importance of any effects that were observed--significant or otherwise.

Also note, not a new issue Same topic treated more completely in 1990s:

http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1225&context=usgsnpwrc

https://pdixon.public.iastate.edu/stat402/readings/cherry98.pdf

The all research is wrong article is not about testing, but about snooping, dredging and hunting for relationships beyond the experimental design or even in the absence of. These exercise are only hypothesis generating. Follow up independent experiments are then needed to test them.

So far only one side of the problem seems to be highlighted and most seem to have ignored the other possible and extremely common way of misinterpreting significant results. In the field of medicine, researchers are in a haste to suggest preventive or remedial measures and they are often carried away by statistical significance. For example, obesity is consistently and significantly associated with insulin resistance across different studies with different ethnic, cultural, nutritional, behavioural backgrounds. A widespread belief therefore is that controlling obesity is sufficient to prevent type 2 diabetes mellitus (T2DM). However, across studies the variance in insulin resistance explained by obesity is very small, the mode being less than 10% and median around 15% (Vidwans and Watve J. Insul Resist 27; 1(1)). Therefore whether controlling obesity alone would be sufficient to prevent T2DM is questionable, but since most people are carried away by the high level of statistical significance, nobody looks at the variance explained and wonders whether and to what extent the association is biologically significant.

This applies to several other associations. The associations of specific food components to obesity, or microbiota to behaviour and health, GWAS studies of obesity or diabetes are all marked by high level of significance but very low variance explained. But researchers are often quick to jump on suggesting translational measures based on high level of statistical significance alone. Unlike statistical significance, biological significance is contextual and purpose driven, and therefore unlikely to be captured in a single number. Nevertheless we need to talk more about biological significance over statistical significance in the field of translational medicine.

Harshada Vidwans, Anagha Pund, Milind Watve

Consider the literature on bioequivalence where biologically meaningful effect sizes are defined a priori. I agree with your comment and note equivalence iscnot a complete fix, but in the right direction.

If I understand correctly, this is a harder problem to solve. For the mean, each observation is a replicate to estimate the population mean. However, I will (or can) use all replicates and I only have one observation as an estimate of the variance. Replicating the variance takes multiple samples of the same sample size, and that is a great deal more work.

Hello

The debate is not limited that question of significance it is more about the question of truth in sciences. P-value is only an indicator and it should be ok to use it or not. If p-value is retired for another totem of truth the situation of sciences overall wil not be improved. Karl Popper was on that point that truth in sciences is only a question of conjectures. This debate on p-value remain that more than everything else and it should be ok to do sciences without statistics.

Sincerly