Statistical Inference in the 21st Century: A World Beyond p < 0.05

https://www.tandfonline.com/toc/utas20/73/sup1?nav=tocList&fbclid=IwAR3Bqo-hUAvrvvpHRfDUezn102TTn0DfOzAZmoG48cBaciIU2aOVJz_EwpM

Useful discussion:

https://www.researchgate.net/post/Retire_Statistical_Significance_A_COMMENT_on_nature_communications

In the application of statistics the outcome is not about truth or proving something. Just because there is a 1 in 20 chance of getting some result does not mean that result did not happen by chance alone. Determining what to do with a p-value < 0.05 is entirely up to you (modified in part by accepted norms).

Statistics is the mathematics of uncertainty. 1+1=2 is easy until I tell you that there is variability about each value. Say I have a drug that will add 1 year to your life (standard deviation of 4 years) and I have another drug that will also add 1 year to your life (standard deviation of 3 years). I have tested each drug on 10 different people (20 people in total) before you and find these results to be true. I offer you both drugs. Do you take them? You ask if others have tried both drugs. I answer yes, four have and have lived an average of 3 years longer.

There could be an interaction effect and so 1+1=3, or this could be a sampling artifact. The problem gets more tricky if I ask if the distributions were normal (Gaussian) or some other distribution. In a sample size of ten, there is no way to tell. What is truth here? Can I know with certainty that taking both drugs will increase my lifespan by 2 years?

Statistics does not fail. It can be abused and misused. It can be used to trick people. However, in any activity with money or power there are individuals and companies that are willing to do anything to scam the system. This is a failing of human nature, not statistics.

There is a very difficult problem to be critical enough to avoid getting scammed while not being so critical that you miss a great opportunity.

Thank you Ayman Amin for suggesting similar discussion and useful link.

Thank you so much Timothy A Ebert for expressing your insight on this issue. I agree with your view but I have a few questions to you, what would be the possible benefits of statistical testing in descriptive cross-sectional studies? If statistics work on the basis of probability and help us only in decision making. You have said " In the application of statistics the outcome is not about truth or proving something " then why we have to waste our time? If it is neither about truth nor prove something. At least it's outcome should be helpful to us. In your view, what are the possible advantages of statistics and how can we implement them?

One needs to be very clear on the goals of the project. If the goal is to describe something then statistics may not be useful or possible. I found an ant in my back yard, I took some measurements, and I want to describe what I saw. However, a description of a single individual is of limited use. It tells me little about what other individuals are like. I might try to describe a new species based on my measurements, but I risk describing males and females as different species (this happened a great deal with ants). Now I have to collect more individuals and then what? The measures of all 500 individuals is simply raw data. I will need to summarize it at least, and I will need to compare it to others. How do I do that?

You don't need an absolute truth for something to be useful. If I could increase world grain harvests by 5% that would be very useful even if there were a few farmers that would experience total crop failure with the new methods.

Statistics helps you make sense of a variable world. If I go and weigh myself can I then conclude that all male humans weigh exactly 87 kilograms? I weigh myself tomorrow and find that I now weigh 86 kilos, and the day after I am at 88 kilos. What is my true weight? If we are all changing weight day by day, why take weights at all? Is there any possibility that the scale that I used could be off by a gram or two? Yet, every time I go to the doctor they take my weight. Why would they do this if there is no possible way that they will know my true weight down to the last atom?

الاحصاءات مفيدة في الاستدلال على النتائج العلمية الدقيقة

The statement

"Scientific inference is more important than statistical inference."

is not a completely valid statement.

Statistical inference and scientific inference are complementary.

The purpose of statistical inference is to achieve scientific inference and scientific inference is achieved through statistical inference.

Dhritikesh Chakrabarty

How could you say that Scientific Inference is achieved through statistical inference? Do you have any basis or you just said because you read like that in statistics book? Could you elaborate your statement through example?

@ Dhirendra Nath ,

Go through the following papers in order to earn your desired idea and concept.

1. Dhritikesh Chakrabarty (2018): “Understanding the Space of Research”, Biostatistics and Biometrics Open Access Journal, (ISSN: 2573-2633), 4(5), 001 – 017: DOI: 10.19080/BBOAJ.2018.04.555642.

2. Dhritikesh Chakrabarty (2018): “Statistics and Bioscience: Association in Research”, Significances of Bioengineering & Biosciences, (ISSN 2637-8078), 2(5), 001 – 007: SBB.000546.2018. DOI: 10.31031/SBB.2018.02.000546.

3. Dhritikesh Chakrabarty: " Association of Statistics with Biodiversity: Study on the Significance of Change in Biodiversity Component", Biodiversity Threats and Conservation, Vol - II (ISBN: 978- 93- 81563- 48- 9), 2014, Global Publishing House, India, pp 193 ‒ 203.

I agree @Dhritikesh Chakrabarty

Dhirendra - 

I would not equate "statistical significance" with "statistical inference." 

But before clarifying that, since "significance" has come up: The word "significance" has taken over the conversation so drastically over the decades that I have trained myself to use the word "substantial" rather than "significant" when I am talking about something tangible, to avoid anyone confusing what I am referencing.  The word "significance" in "statistical significance" is a misnomer.  A stand-alone p-value is virtually meaningless.  But I think many have misinterpreted its meaning because of wishful thinking: that one number could automatically make a decision for you.  A favorite example for me is with regard to heteroscedasticity in regression.  There are "tests" which can be used to say whether or not you have heteroscedasticity.  But heteroscedasticity in regression is a matter of degree.  It is far better to estimate or find a reasonable default for the coefficient of heteroscedasticity, and use weighted least squares (WLS) regression, than to do a "test."  What will a test tell you?  Even if you find a way to decide on a good threshold, based on degree of heteroscedasticity and sample size, it still does not tell you what degree of heteroscedasticity to use if you decide you do have heteroscedasticity, nor how far off you will be if you assume OLS, which is a special case of WLS (and OLS is often not a good default, which is used anyway).

Dhirendra, you said "...what is the importance of statistical testing in the research if it cannot conclude what is the reality? ... Is it the failure of statistics?"  I think that is a good summary of the problem.  Hypothesis tests, or worse, tests of significance, have been expected to "...conclude what is the reality" when they never had that ability.  They were originally, I think, designed, I think Fisher basically said, to tell you if you might want to continue an experiment or not, and provided weak information for that, in my opinion.  But they seem to have taken on a life of their own.  I found very little if any use for them in my work since about 1990.  I think more in terms of estimation, prediction, bias and variance.  Graphics can be very helpful. 

Now, as for "Statistical Inference vs Scientific Inference," we first need to agree on what those terms mean. 

Statistical inference means being able to say something about a population, based on a sample.  Say you want to estimate the mean (or 'predict' it through regression).  Or perhaps you have an experiment, and you want to use statistics to infer general results for all cases, based on that experiment.  A p-value might be part of that, but just a part, if that.  (What about a type II error probability analysis, or effect size?)  Couldn't a confidence interval or a prediction interval be used instead? 

But scientific inference appears to me to be a much broader term.  As a statistician I'll say that statistics are just tools to help make sense of data.  John Tukey said that "The best thing about being a statistician is that you get to play in everyone’s backyard."  (See https://todayinsci.com/T/Tukey_John/TukeyJohn-Quotations.htm.)  So statistics can be useful in virtually every field.  However, it seems to me that "scientific inference" could include inventing a new kind of glue for a special purpose, based on ideas gleaned/inferred from other experiments.   Or one could infer that, based on the current level of scientific research and development/knowledge, driverless cars, "substantially" safer than human driven cars, will be a reality in three years.  Scientific inference can sometimes perhaps be a subjective term. 

I think perhaps the biggest problem is to define terms so that we are all talking about the same things.  In another situation, someone on a thread run by the American Statistical Association recently told me it was too pedantic for me to expect we not use the same word for somewhat different purposes.  But I think that can be problematic.

Cheers - Jim

Mohammad A. Fawzy, why do you agree with Dhritikesh Chakrabarty ? do you have any own view on this matter or you are agreeing just you had also read like that in statistics book?

Thank you so much James R Knaub for distinctive insight.

In my opinion, Statistical Inference is essentially obtained based on a statistical method (tool); it may or may not lead to a Scientific Inference, or sometime may lead to a result that conflicts with a Scientific Inference that is obtained based on a physical law. For example, the well-known t-interval method for measurement uncertainty estimation leads to the results that conflict with the -1/2 power law, the physical law for the relationship between the measurement precision and the number of observations. Therefore, to avoid pitfall of Statistical Inference, we should always consider physics first, and mathematics or statistics second.

Thank you Hening Huang for your insight but my concern is toward the applicability of statistics. Nowadays, there has been increasing trend of quantitative research conduction and publication in developing countries. Specifically in social sciences if the outcome of the research are not implied on policy then what are the benefits of statistics on those surveys. These figure will further change along with time.

Dhirendra Nath -

Above you seem to be concerned about the "...applicability of statistics. ... Specifically in social sciences if the outcome of the research are not implied on policy then what are the benefits of statistics on those surveys."

I am not certain of your meaning, but if you mean that there is a disconnect between what one needs to know, and what a survey is found to learn, then I see two reasons for that. First, when a statistician works with subject matter experts, the first, and arguably most important job is to understand the goal of the project. I suspect that often people do not really know what their goal(s) is (are) in specific terms until they might actually be questioned about it, so that the right information can be gathered. Otherwise, results can not be satisfactory, and perhaps even irrelevant.

But if your concern is not that, but that results are contradictory to policy derived by some other process, then either there is a problem with the data, or a problem with that other process. Statistics are used to help learn, and if the information gathered is relevant, there should be no contradiction.

Jim

The basic jobs to be done in statistical Science are

(1) Collection of Data,

(2) Analysis of Data

and (3) Interpretation of Results/Findings.

Analysis of data consists of two broad jobs namely "Estimation (Statistical Estimation)" and "Testing (Statistical Testing) of Hypothesis". This two is together often termed as Statistical Inference.

Interpretation of of Results/Findings and then drawing of conclusion(s) are possible only after Testing (Statistical Testing) of Hypothesis.

Thus, testing of hypothesis is the basis of drawing conclusion/inference in any research.

In a research study, the objective to be achieved is framed in statistical statement,

Then a statistical hypothesis is set up for testing.

After testing is over, the conclusion/inference is drawn on the basis of the result(s) of the test. The conclusion/inference drawn so is in the statistical literature. Accordingly, this conclusion/inference is regarded as statistical inference.

In the next step, the statistical inference drawn is translated in the literature, of the corresponding study of research, which can be regarded as scientific inference.

Thus, scientific Inference is achieved through statistical inference.

There is a wide agreement among the statistician community that Null Hypothesis Significance Test (NHST) is a never-justified practice that originated from confusions in the early history of modern statistics (see for example, Article Why statisticians are abandoning statistical significance

Article Mindless Statistics

“Most scientific research is exploratory in nature.” The formal statistical inference can only play a limited role in scientific inference. It is statistical thinking enables good science. (references: Article Statistical Inference Enables Bad Science; Statistical Think...

Article The Limited Role of Formal Statistical Inference in Scientif...

@ Gang (John) Xie ,

I don't agree with the opinion that Null Hypothesis Significance Test (NHST) is a never-justified practice. Hypothesis testing is purely based on scientific logic/philosophy and hence is a justified practice..

It may be acceptable that statistical inference can play a limited role in scientific interface. However, this role is too significant to make the conclusion(s) and/ or to draw inference and/or to interpret results in analysis of observations / data in obtained from experiments / survey.

It is to be noted that the role of every branch of science is limited within some domain.

Hi Dhritikesh,

Thanks for your care about my view on the NHST. May I recommend you to read the very interesting and readable book David Salsburg (2001). The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century. Henry Holt and Company. and an excellent article on how the NHST came into being as a hybrid paradigm of Fisher's test of significance and Neyman-pearson's hypothesis test: https://oxford.universitypressscholarship.com/view/10.1093/acprof:oso/9780195153729.001.0001/acprof-9780195153729-chapter-13 .

Furthermore, you may find a list of literature on NHST here: Technical Report LiteratureListOnNHST

1. Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar (2019). Moving to a World Beyond “p

If we abandon NHST, what we lose?

Hening Huang

Hi Hening,

Since NHST is logically not defensible, technically flawed, and practically damaging (to scientific research), by abandoning NSHT we lose nothing. Instead, we should ask what we gain if we abandon NHST. There are tons of literature on the fundamental flaws of NHST - see for example Technical Report LiteratureListOnNHST

I agree, NHST is a very complex issue and to be able to appreciate what I wrote here requires a good understanding of mathematical statistics and careful reading of the relevant literature. I really suggest you to start from reading Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar (2019). Moving to a World Beyond “p

Dhritikesh Chakrabarty

Just to add a few more details of my view on NHST.

The currently most popular statistical inference device, Null Hypothesis Significance Test (NHST) is the result of the anonymous merger of two fundamentally different statistical testing paradigms (Fisher’s evidence-oriented significance testing (regarding the inferential interpretations about the truth of a nullifiable hypothesis) and Neyman-Person’s behaviour-oriented hypothesis testing (regarding a decision-making procedure about two competing hypotheses)) into one seemingly unified/universal testing procedure regarding the so-called ‘statistical significance’.

Classic textbooks sell a unified approach of statistical inference, in particular, the NHST was presented as the indispensable tool, as the sine qua no of scientific research. However, “Unlike its statistical inference counterpart, the concept of scientific inference defies reduction to a series of allegedly neat and tidy methodological steps whose dutiful observance renders the output “science.” Believing otherwise is wishful thinking.” (quoted from Raymond Hubbard, Brian D. Haig, and Rahul A. Parsa (2019), The Limited Role of Formal Statistical Inference in Scientific Inference, The American Statistician, Vol. 73, No. S1, 91-98.)

Most scientific research is exploratory in nature. The formal statistical inference can only play a limited role in scientific inference. It is statistical thinking enables good science. (C. Tong, 2019 TAS Vol. 73 No. S1, 246-261).

Context is king in statistics. And the seven words principles for performing statistical analysis without referring to statistical significance are: “Accept uncertainty. Be thoughtful, open, and modest.” (Ronald L. Wasserstein, Allen L. Schirm& Nicole A. Lazar (2019), Moving to a World Beyond “p

". . . then what is the importance of statistical testing in the research if it cannot conclude what is the reality?"

The p-value from statistical testing can often be transformed into quantities relevant to drawing meaningful conclusions.

For example, the probability that you correctly estimated the sign or direction of the effect can be estimated by plugging the p-value into this simple formula:

https://bityl.co/6Tco

Much has been said in the literature about why NHST in specific and statistical inference in general cannot conclude what is the reality. To cut the long story short, I am with W. Edwards Deming by quoting what he said about NHST. Deming and many other statisticians have rejected the use of hypothesis outright. Deming said: "The student should avoid passages in books that treat confidence intervals and tests of significance, as such calculations have no application in analytic problems in science and industry." (W. Edwards Deming, Out of the Crisis, page 639.) By 'analytic study', Deming refer it as "in which action will be taken on the process or cause-system... the aim being to improve practice in the future...interest centres in future product, not in the materials studied." To Deming, a finding of a significant difference means nothing. It is the degree of difference found that is important. Statistical inference is based on random sampling distribution of the sample statistics. However, most (if not all) of the real life data are not random samples. As Freedman put it: "Statistical inference with convenience samples is risky. Real progress depends on a deep understanding of how the data were generated. No amount of statistical maneuvering will get very far without recognizing that statistical issues and substantive issues overlap." and "Statistical technique is seldom an adequate substitute for substantive knowledge of the topic, good research design, relevant data, and empirical tests in diverse settings."(David A. Freedman (1991), Statistical Models and Shoe Leather, Sociological Methodology, Vol. 21, pp. 291-313) As far as I can tell, the effort for figuring out "the probability that you correctly estimated the sign or direction of the effect can be estimated by plugging the p-value into this simple formula" is just another attempt to "substitute intellectual capital for labour" (David A. Freedman) for scientific inference. However, “Unlike its statistical inference counterpart, the concept of scientific inference defies reduction to a series of allegedly neat and tidy methodological steps whose dutiful observance renders the output 'science'. Believing otherwise is wishful thinking.” (Raymond Hubbard, Brian D. Haig, and Rahul A. Parsa (2019), The Limited Role of Formal Statistical Inference in Scientific Inference, The American Statistician, Vol. 73, No. S1, 91-98.)

Frank Yates (12 May 1902 – 17 June 1994), in 1931 he was appointed assistant statistician at Rothamsted Experimental Station by R.A. Fisher. In 1933 he became head of statistics when Fisher went to University College London. Yates later became a Fellow of the Royal Society and he was one of the pioneers of 20th century statistics. This is what Frank Yates wrote in 1951 (https://www.jstor.org/stable/2280090): “The emphasis on tests of significance, and the consideration of the results of each experiment in isolation, have had the unfortunate consequence that scientific workers have often regarded the execution of a test of significance on an experiment as the ultimate objective. Results are significant or not significant and that is the end of it.

Research workers, therefore, have to accustom themselves to the fact that in many branches of research the really critical experiment is rare, and that it is frequently necessary to combine the results of numbers of experiments dealing with the same issue in order to form a satisfactory picture of the true situation. This is particularly true of agricultural field trials, where in general the effects of the treatments are found to vary with soil and meteorological conditions. In consequence it is absolutely essential to repeat the experiment at different places and in different years if results of any general validity or interest are to be obtained. In such circumstances a number of experiments of moderate accuracy are of far greater value than a single experiment of very high accuracy.”

David Amiel Freedman (5 March 1938 – 17 October 2008, a Ph.D. from Princeton University supervised by William Feller) was Professor of Statistics at the University of California, Berkeley. He was a distinguished mathematical statistician. Freedman wrote

"Generally, replication and prediction of new results provide a harsher and more useful validation regime than statistical testing of many models on one data set. Fewer assumptions are needed, there is less chance of artifact, more kinds of variation can be explored, and alternative explanations can be ruled out. Indeed, taken to the extreme, developing a model by specification tests just comes back to curve fitting-with a complicated set of constraints on the residuals.” (David A. Freedman (1991), Statistical Models and Shoe Leather, Sociological Methodology, Vol. 21, pp. 291-313)

In his 'Statistical Modeling, Causal Inference, and Social Science' blog Andrew Gelman (the first author of Bayesian Data Analysis) wrote: "...if you’re going to give advice in a statistics book about data collection, random sampling, random assignment of treatments, etc., you should also talk about repeating the entire experiment. ...I don’t know that I’ve ever seen a statistics textbook recommend repeating the experiment as a general method, in the same way they recommend random sampling, random assignment, etc." (https://statmodeling.stat.columbia.edu/2020/02/14/42221/ , accessed on 18/11/2020)

Therefore, another dirty secret that the statistics textbooks never intend to tell us is, statistical inferential analysis based on a single set of sample data to its best is what-if analysis, by no means of confirmatory. Experimental design with randomisation and duplication is necessary for minimizing the confounding effects. However, experimental design is just a necessary condition, not a sufficient condition, for making a generalization of the data analysis outcomes.

“Whatever the statistics show, it is fine to suggest reasons for your results, but discuss a range of potential explanations, not just favoured ones. Inferences should be scientific, and that goes far beyond the merely statistical. Factors such as background evidence, study design, data quality and understanding of underlying mechanisms are often more important than statistical measures such as P values or intervals.” (Valentin Amrhein, Sander Greenland, Blake McShane, Retire statistical significance, Nature(2019), Vol. 567, 305: Comment.)

"Statistics education will require major changes at all levels to move to a post “p < 0.05”world." "The change that needs to happen demands change to editorial practice, to the teaching of statistics at every level where inference is taught, and to much more. However…" (Ronald L. Wasserstein, Allen L. Schirm& Nicole A. Lazar (2019), Moving to a World Beyond “p

Dhritikesh Chakrabarty

I agree with you that "The basic jobs to be done in statistical Science are

(1) Collection of Data, (2) Analysis of Data and (3) Interpretation of Results/Findings. Analysis of data consists of two broad jobs namely "Estimation (Statistical Estimation)" and "Testing (Statistical Testing) of Hypothesis". This two is together often termed as Statistical Inference." However, I don't think (and do not agree) "Interpretation of Results/Findings and then drawing of conclusion(s) are possible only after Testing (Statistical Testing) of Hypothesis. Thus, testing of hypothesis is the basis of drawing conclusion/inference in any research." I personally prefer "estimation" other than "significance testing". Scientific inference must be based on parameter estimation that may lead to an estimated whole picture of the quantity of interest, e.g. "effect size". Scientific inference can not be made merely based on the p-value (e.g. from a t-test) that can always be made less than 0.05 when the samples sizes are sufficiently large.

If you think statistics is just roaming around old relationships. I invite you to check out my work. I invented Almost Sure Hypothesis Testing 5 years ago, which says one should let the significance level go to zero at an appropriate rate. It also resolved the Jeffreys-Lindley Paradox. Confidence intervals are usually equivalent to a hypothesis test whenever an inference is drawn from them. I also just proved the multivariate DKW inequality, which has a beautiful form which was thought to be impossible. One can demonstrate my DKW inequality is valid by using an almost sure hypothesis test with simulated data. I didn't do that in the paper. Nature is not a statistical Journal and it should not be cited as the final word. Most of the people that sign these petitions are social scientists, not statisticians like me.

No hypothesis is proven false with a single small p-value, nor is it proven true with a large one. All we can do is provide the weight of the evidence. I find confidence distributions to be incredibly helpful for making this point and visualizing inference.

Preprint Decision Making in Drug Development via Inference on Power

Preprint Tolerance and Prediction Intervals for Non-normal Models