The debate on null hypothesis significance testing (NHST) and p-values has been going on for decades and is still going on. One side of the debate advocates abandoning NHST and p-values and promoting statistics reform (new statistics), while the other side defends NHST and p-values.
From my observations, many scientists want to abandon NHST and p-values, while many statisticians do not, although there are a few exceptions on both sides. Table 1 shows twelve distinguished scientists who advocate abandoning NHST and p-values. Table 2 shows fifteen statisticians who are the co-authors of the article: “ASA president’s task force statement on statistical significance and replicability”, which defends NHST and p-values.
As can be seen from Table 1, these scientists include economists, psychologists, physiologists, zoologists, clinical epidemiologist, and ecologists. They are all practitioners who use statistical methods (tools) in their work or practice. On the other hand, as can be seen from Table 2, most of these statisticians are university professors who teach statistics and manufacture (or invent) statistical methods (tools).
So, what do you think of this situation: “Why do many scientists want to abandon NHST and p-values, but many statisticians don’t?”
Table 1 Twelve distinguished scientists who advocate abandoning NHST and p-values
Stephen T. Ziliak， Professor of economics at Roosevelt University, The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives, 2008,University of Michigan Press
Deirdre Nansen McCloskey，Economist at Cato Institute in Washington, D.C，The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives, 2008, University of Michigan Press
David Trafimow, Social Psychology, Professor at New Mexico State University, Editor of Basic and Applied Social Psychology, Banned NHST in Basic and Applied Social Psychology since 2015
Tom Siegfried, Editor in Chief of Science News, To make science better, watch out for statistical flaws, 2014, Science News
Thomas Heckelei, Economist, Professor at University of Bonn, The p-value debate and statistical (Mal) practice–implications for the agricultural and food economics community, German Journal of Agricultural Economics, 2023, 72(1), 47-67
Lewis G Halsey, Professor of environmental physiology at University of Roehampton, The reign of the p-value is over: what alternative analyses could we employ to fill the power vacuum? Biology Letters, 2019, 15(5), 20190174
Mark Elkins, Clinical Associate Professor at University of Sydney, Statistical inference through estimation: recommendations from the International Society of Physiotherapy Journal Editors, European Journal of Physiotherapy, 2022, 24(3), 129-133
Geoff Cumming, Retired after a lifetime of teaching psychology at La Trobe University in Melbourne, The New Statistics, Psychological Science, 2014, 25(1)
Valentin Amrhein, Professor of zoology at the University of Basel, Why and how we should join the shift from significance testing to estimation, J Evol Biol. 2022, 35(6), 777-787
Wenjun Zhang, Professor at International Academy of Ecology and Environmental Sciences, Sun Yat-sen University, A desktop calculator for effect sizes: Towards the new statistics, Computational Ecology and Software, 2023, 13(4): 136-181
Norbert Hirschauer, Professor of agribusiness management at the Martin Luther University Halle-Wittenberg, Some thoughts about statistical inference in the 21stcentury, SocArXiv. December 20 2022, doi:10.31235/osf.io/exdfg
Stefanos Bonovas, Clinical epidemiologist, biostatistician and research methodologist at Humanitas University, On p-values and statistical significance, Journal of Clinical Medicine, 2023, 12(3), 900
Table 2 Fifteen statisticians who are the co-authors of the article: “ASA president’s task force statement on statistical significance and replicability” (Ann. Appl. Stat. 2021, 15(3): 1084-1085), defending NHST and p-values
Yoav Benjamini, Professor of Applied Statistics at the Department of Statistics and Operations Research at Tel Aviv University
Richard D. De Veaux, Professor of Statistics in the Department of Mathematics and Statistics at Williams College
Bradley Efron, Professor of Statistics and Biostatistics, Department of Statistics and Department of Biostatistics, Stanford University
Scott Evans, Professor and Founding Chair of the Department of Biostatistics Bioinformatics and the Director of the Biostatistics Center at George Washington University’s Milken Institute School of Public Health
Mark Glickman, Senior Lecturer on Statistics at the Harvard University Department of Statistics, and Senior Statistician at the Center for Healthcare Organization and Implementation Research, a Veterans Administration Center of Innovation
Barry I. Graubard, Senior Investigator in the Biostatistics Branch at the National Cancer Institute
Xuming He, Professor of Statistics, University of Michigan
Karen Kafadar, Professor and Chair of Statistics, University of Virginia
Xiao-Li Meng, Professor in the Department of Statistics at Harvard University
Nancy Reid, Professor of Statistics at the University of Toronto
Stephen M. Stigler, Professor Emeritus at the Department of Statistics of the University of Chicago
Stephen B. Vardeman, Professor, Departments of Statistics and Industrial and Manufacturing System Engineering, Iowa State University
Christopher K. Wikle, Professor and Chair of Statistics at the University of Missouri
Tommy Wright, Research Mathematical Statistician and Chief of the Center for Statistical Research and Methodology, U.S. Bureau of the Census
Linda J. Young, Chief Statistician and Director of Research & Development at the National Agricultural Statistics Service
I am not sure where this discussion should lead, but I think that the formulation is already problematic. There is a difference between abandoning NHST and p-values. Where p-values seem to be a legit statistic, NHST seems to be a flawed concept, which uses p-values. But this is not equatable.
Rainer Duesing
I agree with your point: “Where p-values seem to be a legit statistic, NHST seems to be a flawed concept, which uses p-values”. It is true that the p-value is a neutral number that cannot be good or bad. However, the problem is that p-value is produced by NHST that is logically flawed and its interpretation using the NHST language leads to so much confusion and misconceptions. Therefore, if we agree to abandon NHST, p-value should be abandoned at the same time. I don’t see any other way, i.e. abandon NHST without abandoning p-values.
I think it's a matter of theory versus practice. Of course all good practice has to have good theory behind it. But it struck me, perhaps 40 years ago, that it was tempting for a theoretical statistician to love developing the distributions for null hypotheses, regardless of any practical usefulness. A p-value may be well-defined, but a lone p-value is not useful. This all prompted me eventually to write the following:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
Today I think it more useful to estimate "how much," rather than look for a yes/no answer. For example, there are hypotheses tests for heteroscedasticity in regression. (There really should be heteroscedasticity if predictions are of different sizes.) But what do you do if you "conclude" that "yes," there is heteroscedasticity? You can use weighted least squares and you can estimate the coefficient of heteroscedasticity or use a reasonable one. If you conclude "no," there isn't any, then what if a different test says "yes." You may realize it isn't likely to be exactly a case of homoscedasticity. What does "yes" or "no" really mean? How close is it? Why bother with such a test? Learning the nature of the data and how well you can model it, and using weighted least squares regression, doesn't require this.
Hening Huang that is where we may disagree. I would not ban p-values, but put them into more context. As I already said in another thread, I am fond of Bayesian statistics, but I am pretty sure, this can and will be exploited as well, probably as any other statistics (that's why I don’t like the Bayes Factor, where we also find arbitrary cut-offs and my further explanations will also concern this).
My biggest problem is this dichotomous decision criterion. I think this has been so immensely exploited, since researcher (or maybe all humans...) look for simple explanations of complex problems. For this purpose, a simple, dichotomous criterion seems to be perfect, you do not need to think further. This saves time and energy. In my opinion, this is the main reason, why it is so popular and so hard to get rid of it.
If I would get a penny for every time a student said "I want to test, if my difference/correlation/slope ist significant.", I would be rich. They don’t learn to think further, that a significant result does not answer their original questions, for example, if the intervention they investigate, is of any help. And this is even true, if they know about confidence intervals, effect sizes and so on. It is simply so simple to apply.
Therefore, in my personal view, p-values are not the problem, but the dichotomous decision of NHST (and how it is taught, and no, I do not mean the NP approach. Nobody really calculates the loss and alternative hypothesis in psychology….). And yes, I know you also mentioned in other threads that p-values do not give you THE answer and I would totally agree. BUT a continuously interpreted p-value, along with confidence intervals, unstandardized and standardized effect sizes, descriptive statistics, visualization of the data and maybe other metrics (as you have proposed in one of your articles), we can get a good feeling, what our results mean and evaluate the practical relevance.
So, to answer your question: I think this is also doubly wrong dichotomy. There is not this clear cut between scientists and statisticians AND p-values and NHST aren't the same. BUT what I could think of and goes into the direction you were leading, is that scientists probably want to focus more on the effectiveness of the approach, whereas statisticians focus more on the correctness of the concept itself.
And if you have a look at the ASA President’s Task Force Statement, they do not say to rigorously apply the 5% rule, do dichotomous decisions and forget about everything else. For example, they conclude in the end:
"In summary, P-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data. Analyzing data and summarizing results are often more complex than is sometimes popularly conveyed."
And some paragraphs before:" Different measures of uncertainty can complement one another; no single measure serves all purposes. The sources of variation that the summaries address should be described in scientific articles and reports. Where possible, those sources of variation that have not been addressed should also be identified."
So, I read it in a way that you should not solely rely on p-values and "statistical significance", but look at all information available. Hence, scientists and statisticians seem not to disagree very much. But I would love to hear other opinions.
P.S.: From a practical point of view, it is much harder to get rid of NHST and p-valus only reporting, I know that. Where I was able to convince a lot of people around me to explore the data further, to understand them, it is a whole other story to "sell" this to journals, editors and reviewers!! I started to pack it into supplementary materials, so that it is available to the interested readers, but it is very hard, at least in psychology, to do it differently!
P.P.S.: I am a psychologist and I have to admit that especially psychology seems to have a big problem: I have the impression that a lot of researchers do not know what their dependent variables mean, how to interpret them properly and hence how to evaluate changes or differences on that dependent variable. This is sad and frightening, but fact. Constantly, new questionnaires and tests are developed, but nobody knows what they really mean. They use standardized scales and Likert items, but do not think about the meaning and validity. Therefore, all they can do is to look at the p-values and „statistical significance“, because they are unable to evaluate the practical relevance. They simply do not know what they are doing (and I was guilty of this as well, but I try to do better)…. this is a bit exaggerated, but hits the nail on the head somehow.
Rainer Duesing
I agree with you that “dichotomous decision criterion” is a big problem. However, even if we do not use a “dichotomous decision criterion” with p-values, and instead use continuous p-values for decision-making, there are still three problems with p-values: (1) p-value is calculated using a logically flawed procedure NHST, and therefore its validity is questionable, (2) the meaning (mathematical or physical) of p-value never be clear for any practical problems, and (3) p-value is a function of sample size; a large sample will always produce a small p-value even if the effect size is very small and meaningless, which is misleading. Yes, people can argue that we do not rely on p-values alone and we always report effect size and look at other statistics. However, as I argued in another discussion, “… the p-value at best is an indirect measure of uncertainty in the estimated effect size, and as Alessandro Rovetta puts it: “The P-value is (finally) an inherently uncertain measure” (---). But why bother using this indirect measure that causes so much confusion and misinterpretations when we can directly measure the uncertainty using “standard error” or “standard uncertainty” as we do in measurement science?”
Statistical methods are tools for scientific research. If we know a tool is flawed, why should we still use it? NHST is a flawed tool. We can use other good tools (e.g. estimation statistics) instead of NHST. NHST and the related p-values have been shown to do more harm than good in science. That is why so many scientists want to get rid of them. I think scientists can do research without using NHST and p-values.
It would be pity if “… a lot of researchers do not know what their dependent variables mean … all they can do is to look at the p-values and „statistical significance“, because they are unable to evaluate the practical relevance”. Scientists should reply first on their domain knowledge, not statistical methods. In other words, “domain knowledge” should always come first and “statistical methods” second, not the other way around.
Hening Huang
It is discouraging to see that after all these discussion you still are bound to these misunderstandings that you nicely summarize here again:
ad (1) The calculation of p-values has nothing to do with NHST. There is nothing flawed in their caclulation.
ad (2) The "(physical) meaning" of p-values is the same as of probability in general. There is no such meaning. It is often stated that the (physical) meaning of probability is a long-term relative frequency, but this is flawed (see the https://joelvelasco.net/teaching/3865/hajek%20-%20fifteen%20arguments%20against%20hypothetical%20frequentism.pdf)
ad (3) This is exactly what the p-value was invented for - to in some was quantify the the amount of information provided by the observed data. As every obsevation adds the same expected amount of information, this must be a function of the sample size.
Jochen Wilhelm
To clarify, I never said that the calculation of p-values was flawed. What I emphasized was that p-value is the result of (or calculated based on) the logically flawed NHST. I am surprised by your statement that “The calculation of p-values has nothing to do with NHST”. Please show an example of a p-value that is not calculated based on a NHST. Thank you.
Hening Huang
correct me if I am wrong (or if you have another view), but already the Fisherian approach should not be considered as NHST. Colloquially, Fisher, NP and NHST are sometimes called or subsumed under "NHST", but this is really not the same.
Hening Huang ,
as Rainer already mentioned: tests of statistical significance came first. This was formalized by Fisher in the 1920s. Up to the 1940s there was a (yet unresolved) fight between Neyman and Fisher about the neccesity of an alternative hypothesis (so that a real decision is possible rather hat just opposing "sufficiently conclusive" from "not sufficiently conclusive"). Neyman developed a behavioral algorithm to control a priori fixed error rates on decisions between two distinctively different alternatives A and B, for which actually only the acceptance regions for the test statistic had to be calculated - not a p-value). These to testing incompatible principles were combined in a illogical way in what is called NHST, where the "alternatives" are defined as "null hypothesis" vs. "not-null hypothesis", and a decision between these two is thought to be provided by a low p-value, which in addition should also keep some kind of error rate at alpha if the alternative is accepted only when p < alpha.
As an even more early exapmle of the reasoning behind statistical significance test you may have a look at
‘An argument for divine providence, taken from the constant regularity observ’d in the births of both sexes’, Philosophical Transactions (1683–1775) 27, 186–190.
Jochen Wilhelm (and Rainer Duesing )
I know there is a difference between Fisherian significance testing and NP’s hypothesis testing, and that NHST is a hybrid of the two approaches. In any case, there is no example showing that a p-value can be calculated without NHST or significance testing.
Let’s move to your comment: “ad (3) This is exactly what the p-value was invented for - to in some was quantify the amount of information provided by the observed data. As every observation adds the same expected amount of information, this must be a function of the sample size.” My original statement is copied here “(3) p-value is a function of sample size; a large sample will always produce a small p-value even if the effect size is very small and meaningless, which is misleading.”
In response to your comment, let me quote Trafimow (2023) and Hirschauer (2022) below.
Trafimow (2023) stated,
“… if we concede that researchers are better off carefully perusing the descriptive characteristics of their data, not just with respect to summary statistics but with respect to distribution shapes too, then it seems unavoidable that the confounding is deleterious. To reiterate, p-values provide no added value to a careful perusal of the data. Confounding sample effect sizes and sample sizes, in the form of p-values, does not gain the researcher anything above and beyond keeping them distinct and closely attending to the data.
Hirschauer (2022) stated,
“What we can extract – at best – from a random sample is an unbiased point estimate (signal) of an unknown population effect size and an unbiased estimation of the uncertainty (noise), caused by random error, of that point estimation, i.e., the standard error, which is but another label for the standard deviation of the sampling distribution. We can, of course, go through the mathematical manipulations to transform these two pieces of information into a signal-to-noise ratio (z- or t-ratio), a p-value or even a dichotomous significance statement. But why should we do that and accept the loss of information and risk of inferential misinterpretations?”
Hirschauer N 2022 Some thoughts about statistical inference in the 21st century SocArXiv. December 20 doi:10.31235/osf.io/exdfg
Trafimow D 2023 The Story of My Journey Away from Significance Testing A World Scientific Encyclopedia of Business Storytelling, pp. 95-127 DOI: 10.1142/9789811280948_0006
Hening Huang
now you write that "there is no example showing that a p-value can be calculated without NHST or significance testing" (emphasis mine). Of course not, why should one calculate a p-value if it is not used for anything? The p-value is the basis of significance testing and it is used in NHST (although the latter could be achieved by defining acceptance regions, since the desired "type-I error" is fixed.
ad Trafimow: I agree. Significance testing is not very sensible for large samples (more concretely: for models with a large number of degrees of freedom). If there is so much information available, it can be used much better than just doing a significance test of a (presumably) too simple model.
ad Hirshauer: I do not agree. Significance tests should be a safety elt for us to avoid interpretations of patterns we might see in data sets too small to provide sufficient information about these assumed patterns. It's a safe-guard against too many misinterpretations. Sure, if applied wrongly, no kind of safety belt is effective and can even be harmful. But again, this is to blame to the wrong use, not to the safety belt.
Hening Huang
it seems a little bit, like you are shifting the goal post. In your first answer you demanded an example, where p-values are used/calculated without NHST. After giving you the Fisher example, you demand to see p-values without NHST or significance testing. Why? Your main argument was against NHST. What is your argument against significance testing, now? We already agreed that NHST is flawed.
You may have your personal preference to dislike p-values and don't use them (I also would agree and try to do it otherwise), but personal feelings do not change the correctness or general applicability.
Rainer Duesing
To clarify, to me and to many authors (e.g. Herbert 2019, Vinceti and Filippini 2021, Gorard 2017), null hypothesis significance testing (NHST) and significance testing are the same thing. Therefore, like many authors, my argument is against NHST/significance testing.
Herbert R. (2019). Research Note: Significance testing and hypothesis testing: meaningless, misleading and mostly unnecessary. Journal of Physiotherapy, 65(3):178-181. doi:10.1016/j.jphys.2019.05.001. Epub 2019 Jun 4. PMID: 31175039
Silvio Roberto Vinceti & Tommaso Filippini (2021). Towards the dismissal of nullhypothesis/statistical significance testing in public health, public law and toxicology, Public Health Toxicol2021;1(2):7, https://doi.org/10.18332/pht/144290 .
Stephen Gorard (2017). Significance Testing is Still Wrong, and Damages Real Lives:A Brief Reply to Spreckelsen and Van Der Horst, and Nicholson and McCusker, Sociological Research Online, Volume 22, Issue 2
I cannot claim to be a technical expert on this issue. However, as someone who worked as a statistician for a long time, guess I am qualified to add a comment. Many times I had resorted to hypothesis testing and as a practitioner, found the tool extremely useful, especially in the old days when we used statistical tables. I also liked the conceptual simplicity. The concept of p-value apparently comes from Gauss and Astronomy. The use of it in Agriculture has been questioned by Prof S.C. Pearce since the late last century. As Prof. Fisher himself pointed out, researchers ask research questions from the data and all appropriate statistical outcomes should be considered in this spirit. Thus, can the key issue here be not the use of significance testing itself, but the blanket use of them singularly without properly looking at the data and the outcomes of testing to engage the researcher's logical reasoning? I often found significance testing to be very helpful in practice. That doesn't mean it is always smooth sailing. A problem I encountered with the interpretation of two sample tests was discussed in Technical Report Some Practical Considerations about the t-Test for Two Indep...
Darshi. Arachige
Your technic note reveals an important philosophical problem with the two-sample t-test. I agree with your view: “With the t-test for two independent samples, we are not in the same territory as we are with the statistical induction from the experiment on the lady tasting tea. There we had a clear hypothesis and a clearer probabilistic approach to test the hypothesis. With the t-test, we depend heavily on the theoretical distributions.” In my opinion, the two-sample t-test cannot provide a valid solution to practical problems. Please refer to the discussion: https://www.researchgate.net/post/Does_the_two-sample_t-test_provide_a_valid_solution_to_practical_problems
Hening Huang
I read your paper on Exceedance Probability with interest. It has been written so well in an easily readable manner. Your proposed approach to the problem of significance testing is especially well-suited when we only have the summary stats. But wouldn't one underlying issue about two sample t-test (independent samples) still remain? How do we view the data overlap arising from the two samples? We only use the realised values to estimate descriptive statistics. There can be many unrealised values within the respective data ranges. Should we only look at the two sample t-test with the observed values in mind, ignoring any possible ergodicity and just assume the values observed are the only realisable values?
Hening Huang I do not know, if this was a topic in any of this discussion, but how do you think about a Bayesian approach to compare for example two independent groups? Wim Kaijser mentioned the approach by Kruschke and his BEST package. He uses a student t distribution as a Likelihood (for robustness), you are able to incorporate your area specific knowledge (with priors) and you get estimates for the population parameters (the posterior), instead of a sample statistic and of course no p-value.
Please find attached an example output, what is delivered by default. Of course, other metrics, like the ones you mentioned in your paper, are possible to calculate, directly from the posterior. This would have the advantage to have again a posterior distribution of the EP or whatever, where you can see the uncertainty of this parameter.
I would like to hear your opinion about it. Thanks in advance.
Darshi. Arachige
Thank you for your interest in my work on exceedance probability. I think the concept of exceedance probability does a good job in accounting for the data overlap arising from the two samples. Indeed, “We only use the realised values to estimate descriptive statistics.” But we then use the sample mean and sample standard deviation as the estimated population parameters, which is what statistical inference or estimation statistics should do. The calculation of exceedance probability is based on the estimated population parameters (mean and standard deviation) using the data we have at hand.
Rainer Duesing
First of all, I am not familiar with the BEST package. I assume that it employs the standard Bayesian formulation and therefore cannot go wrong in the Bayesian framework. However, how it chooses and justifies a prior is always a question for any Bayesian approach. In addition, I thought that the posterior distribution of the effect size (mu1-mu2) (the simple effect size) was sufficient to assess the uncertainty of the effect size. I wonder if the standardized effect size distribution would add any value to the question under consideration.
I am not a big fan of Bayesian approaches. In particular, I am against using Bayesian approaches for measurement uncertainty analysis, as I have discussed in several papers (Huang 2018, 2020a, b, 2024). Nevertheless, I think the Bayesian philosophy is valid and useful. To me, Bayesian approaches provide a mechanism to combine all avaible information in inferences or decision-making. However, frequentist methods such as LCD (law of combination of distributions) can do the same job. I showed four examples and compared frequentist methods with Bayesian approaches for measurement uncertainty analysis (Huang 2020b). “In general, Bayesian formulations are much more complicated, and Bayesian computations are much more involved (such as the MCMC sampling) than their frequentist counterparts. The frequentist methods (e.g. LCD, least-squares, and point estimation) can produce equally good or better results with less complicated formulation and less computational efforts than their Bayesian counterparts (Huang 2020b).”
Furthermore, I argue that the Bayes formula for continuous random variables (called the reformulated Bayes Theoremin measurement science) is flawed because it is not a self-consistent operation. I also argue that “in the light of the frequentist-Bayesian transformation rule and the law of aggregation of information (LAI), the frequentist and Bayesian inference are virtually equivalent, so they can be unified, at least in measurement uncertainty analysis (Huang 2022).” For a detailed discussion on this topic, please see my paper (Huang 2022).
Huang H 2018 A unified theory of measurement errors and uncertainties Measurement Science and Technology 29125003 https://doi.org/10.1088/1361-6501/aae50f
Huang 2020a Comparison of three approaches for computing measurement uncertainties Measurement 163https://doi.org/10.1016/j.measurement.2020.107923
Huang H 2020b Two simple and practical methods for combining prior information with current measurement in uncertainty analysis, accepted for publication in Cal Lab the International Journal of Metrology 27 22-32 available from ResearchGate:https://www.researchgate.net/publication/344502279_Two_simple_and_practical_methods_for_combining_prior_information_with_current_measurement_in_uncertainty_analysis
Huang H 2022 A new modified Bayesian method for measurement uncertainty analysis and the unification of frequentist and Bayesian inference Journal of Probability and Statistical Science 20(1) 52-79
Huang H 2024 A practical two-step procedure for taking into account all available information (prior and current) about influence quantities in measurement uncertainty analysis Accreditation and Quality Assurance 29(2) DOI: 10.1007/s00769-024-01583-0
Hening Huang Thank you very much for your response, Dr Huang. I agree 100% with your comment about inference. No doubt, we use sample statistics to assume theoretical distributions that capture all possible values. The exceedance probability approach too, in my understanding, uses this theoretic path. In sample situations, as you would agree, a lot depends on the realised sample. If we have 2,3,5,6 and 2,3,4,6 as samples in a trivial scenario, we have two sets of means and SDs, we get means 4, 3.75 and biased SDs 1.58, 1.48, respectively. Not exactly the same! In this case, the assumed population parameters can be either of these. My point in the two sample t test issue I discussed was around the sole dependence on realised values to make inferences. We use sample statistics and do the traditional tests. But before coming to a conclusion, isn't it useful to look at the factors like the possible overlap between the observed sample assuming the reliability of the observed range, not solely the means and their standard errors? Even though this additional step would not solve issues about NHST, in my mind, it would be a little more holistic path to conclude using the comparison of means in the independent sample situation.
Darshi. Arachige
The traditional tests, such as the two-sample z-test and t-test, actually compare the two sampling distributions of the mean; they do not (and in fact cannot) examine the overlap between the observed samples. The larger the sample sizes, the narrower the sampling distributions of the mean, the smaller the overlap between the two sampling distributions, and the smaller the p-value. This is the fundamental reason why the p-value can be easily hacked through the “N-chasing”.
Hening Huang
one minor point: the two-sample t/z-test is about the sampling distribution of the mean difference.
one major point: the tests are to check if that sampling distribution is narrow enough to distinguish an estimate from a hypothetical value. This is a relevant and necessary feature of the test. The test should demonstrate that the sample size is sufficient. Using large samples should make sure that that the sample estimate is distinguishable from a almost all possible hypothetical values. This is not "N-chasing", this is the core of the test. "N-chasing", in contrast, is to steadily increase the sample size until the p-value is small enough. Without appropriate correction (you may google for "alpha spending") of this is a flawed strategy leading to meaningless p values.
Jochen Wilhelm
This is where we disagree. First, I think that now that we have admitted that the NHST is logically flawed, the NHST language may no longer be convincing. Second, I argue that the fact that the t-test and its produced p-value are a function of sample size is an inherent flaw of the method, which leads to p-hacking via “N-chasing”. Chén et al. (2023) considered p-hacking to be a “paradox”. They offered several suggestions to avoid p-hacking and stated,
“Indeed, given unlimited resources, most people may prefer studies with very large sample sizes because they feel larger sample studies are more reliable than smaller trials. Here, we do not advocate against large-sample studies (which have many advantages, as we see below); rather, we argue that one should treat the p value contextually and avoid being that aggressive scientist.”
In reality, there is nothing to stop scientists from using large samples, and in fact, large samples are preferred whenever possible in any study. Therefore, the problem with p-hacking via “N-chasing” cannot be solved unless the p value-based hypothesis testing is abandoned.
What you describe is again just a wrong use of p-values.
If there is a lot of information (e.g. about the mean difference) available, it should be used. This may be done by testing a quantitatively interesting hypothesis (not zero, but some scientifically interesting non-zero value) or by testing all possible values, that is, to construct a confidence interval as estimate and to judge if this interval is excluding scientifically uninteresting values and including scientifically interesting values.
A large study may reveal a mean difference that is highly statistically significantly different from zero and at the same time that the effect is uninteresting, irrelevant, or negligible. A small study may reveal a mean difference that only a barely statistically significantly different from zero (so one may conclude the sign of the expected difference), but fails to provide information about the relevance or irrelevance of the effect.
Hening Huang Your response, Dr Huang, is so great that it captures the issue in a sentence.
"The traditional tests, such as the two-sample z-test and t-test, actually compare the two sampling distributions of the mean; they do not (and in fact cannot) examine the overlap between the observed samples." To add to this, please let me quote from the following note. Technical Report Some Practical Considerations about the t-Test for Two Indep...
"One of the widely used examples to illustrate the use of the two-sample test is the US vs Japan auto miles per gallon data. As the two-sample t-test showed, the difference between the means of these two samples was very highly significant. One is tempted to conclude from the above result that US autos are more fuel-efficient (see the note below). But does this conclusion hold water? ...... The overlap index of 0.65 for this dataset indicates almost two-thirds of an overlap in the dispersions of two samples. " In the above quote, I, as a very run-of-the-mill practitioner, was wondering if I buy a Japanese car in the US within the study's terms of reference, wouldn't that car be almost like a US made in 65 out of 100 times and still be significantly different? For reasons, I did not discuss, I only considered the t-test. I also did not look at the tail densities.
With respect to "N-hacking" I have a different view. I do not believe any honest researcher would increase the sample size to purely get a higher effect. Wouldn't they thus be more inclined to increase precision and accuracy? Like Jochen Wilhelm I tend to think N-chasing is not equivalent to p-hacking that can also be achieved via other means like trimming. Apologies for the long-winded comment! Also, hope I made sense!
I do not get the take with "N-chasing". As Jochen Wilhelm already pointed out, it is a feature, not a bug that the precision increases with increasing N. It doesn't matter if you calculate the standard error of the mean difference or the confidence interval or in turn the p-value. You can be more confident with larger samples. I cannot see how this is a problem in general. It is not the p-value's fault that this may be abused in any form. And other have also pointed out, even if you abandon p-values, you can "cheat" in another way.
Rainer Duesing I also agree with you on this, Dr Duesing. While I do not deny Hening Huang's concerns about some unscrupulous person's attempt of N-chasing, from a practical point of view, the means can not be guaranteed to remain the same or the variance to keep decreasing with an increasing N. Hence, a guaranteed p-hack may not eventuate. Often, results of such sampling may not stand replication. For some traditionally-trained and frequentist leanings, I used to be rather more comfortable with a larger sample size over a very small sample coupled with a prior. Another tacitly expressed point in my previous comment, is that looking at the sample overlap as a complementary exercise diminishes the effectiveness of any N- chasing. My view on this is naturally biased.
I must join the people who are baffled by the concept of N-chasing. The definition of uncertainty is based on the ratio between the amount of data and the variability of the data. The "amount" of data is, of course, measured by square root of N.
To me, as with Rainer Duesing and Jochen Wilhelm , working to decrease uncertainty by decreasing the standard error of estimate should be the goal of good research practice. What defines an adequate N is not some notion of statistical significance but the achievement of a standard error that we can live with for all practical purposes.
You are referring to means, correct Ronán? Standard errors get smaller with n, but standard deviations do not.
For example, sigma for residuals is a standard deviation. (Since residuals are estimated, this gets messy.)
Also, a population or stratum standard deviation is in contrast to the standard error for the mean.
Indeed, with larger samples we are more confident about the precision of the sample mean as an estimate of the population mean because the standard error (called standard uncertainty in measurement science) for the sample mean decreases with increasing sample size. The sampling distribution of the mean becomes narrower with increasing the sample size because it uses the standard error as its scale parameter (estimated). Mathematically, a z-test or t-test accounts for the overlap between the two sampling distributions of the means; it does not (and cannot) account for the overlap between the two underling population distributions. “N-chasing” reduces the overlap between two sampling distributions of the means. The resulting p-value provides essentially no new information for our analysis and inference, because (as I mentioned) the p-value at best is an indirect measure of uncertainty in the estimated effect size. To understand this, let’s consider two treatments (or two products) A and B. The quantities we are really interested in are (1) the mean difference between A and B (i.e. effect size), (2) the uncertainty of the estimated effect size (i.e. precision), and (3) the probabilistic measure of the difference between A and B in terms of exceedance probability or common language effect size (CLES) or probability of superiority (PS). The p-value produced by a two-sample one-tailed z-test or t-test is not a valid probabilistic measure of the difference between A and B; at best, it is viewed as an indicator of the uncertainty of the effect size. Therefore, the p-value does not add any new information. In contrast, the exceedance probability or common language effect size (CLES) or probability of superiority (PS) adds new information to our analysis and inference. In addition, the t-test relies on the t-distribution. But the t-distribution has its own problems due to t-transformation distortion (Huang 2018) and information loss (Huang 2023).
Huang H 2018 Uncertainty estimation with a small number of measurements, Part I: new insights on the t-interval method and its limitations Measurement Science and Technology 29 https://doi.org/10.1088/1361-6501/aa96c7
Huang H 2023 A minimum entropy criterion for distribution selection for measurement uncertainty analysis Measurement Science and Technology 35(3) 035014 DOI: 10.1088/1361-6501/ad1476
When Hening Huang says that "...the p-value does not add any new information" [by itself] I can relate to that. I'm not familiar with the term "n-chasing," but it seems there are two different takes on it being discussed. (1) If you use a large n just to get a small p-value, that, considered alone, is not helpful. It does not help with the effect size. (If you use a small enough sample size, you might 'accept' almost any standard as satisfied when testing whether a contract has been successfully met, because you cannot reject it. There would not be enough information.) But (2) using a larger n to reduce standard errors can be very helpful. I'll take a standard error over a lone p-value anytime. The former can actually be very helpful, but not the latter.
- Badges
- Science topic