The P value (and the phrase “statistically significant") has been the most abused measures in almost all disciplines of science. Despite decades of warnings from statisticians it still continues to be misused and abused.
Let us embrace the advice given by the many statisticians in this article.
Dear Gudeta Weldesemayat Sileshi, this interesting publication attached may complement your questioning. So, the paper by Valentin Amrhein, Sander Greenland, and Blake McShane has just appeared online as a comment piece in Nature
Gudeta -
Long ago I did try to warn that a p-value is not a stand-alone measure. Consider this:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
I've hardly touched them since. And "significance" is a very misleading term.
People may argue that a p-value threshold may be used for decision making, but seldom is it meaningful to conclude that "yes this is true," or "no it is not." It is better to address the question "How different might these things be?" such as in cases where we can estimate and perhaps provide relative standard errors, or confidence intervals, or prediction intervals for "predictions" from regression.
For example, rather than test if we have "significant" heteroscedasticity, one can estimate the coefficient of heteroscedasticity.
With the ability to collect and process very large data sets now, even accounting for the bias problems with "big data," it is painfully obvious that the practice of using a p-value threshold of 0.01 or 0.05 everywhere is nonsense without considering standard deviations and sample sizes. This has always been true. It is just more obvious now.
Thanks - Jim
Gudeta Weldesemayat Sileshi, sir
*****Misinterpretation and abuse of statistical tests, confidence intervals and statistical power have been decried for decades but yet remain rampant. As these concept need a high attention and time, this high cognitive demand has led to an epidemic of shortcut definitions and interpretations that are simply wrong, sometimes disastrously so — and yet these misinterpretations dominate much of the scientific literature****.
Solution... awaited??
Good point. I think the solutions and alternatives are there. Some of them have been discussed in the paper I posted. I believe estimating effect sizes and their confidence intervals (when correctly estimated) give a better alternative to P values. I agree with you confidence intervals (CIs) may also be misused. Asymmetric CIs and bootstrap CIs are better than the traditional Wald CIs, but these are not commonly used.
It's one of the reasons why I think Bayesian inference is more coherent, principled, and unbiased...
Bayesian analysis has always treated the p-value as an unreliable statistic, and it is one of the arguments for Bayesian analysis.
As I wrote in the response to Dunphy (https://www.europeanscientist.com/en/research/scientists-are-rethinking-statistical-significance/ ), it will help a lot if all scientists learn and understand what a p-value is.
Gudeta, Salvatore, and all -
Today I received an issue of Significance, April 2019, volume 16, issue 2, a joint magazine-like publication of the Royal Statistical Society, and the American Statistical Association. After I had published a letter in there, in June 2016, I had suggested to the editor that "significance," like other examples I gave, was a problematic term, regarding which I could write another contribution.
In the case of "significance," I actually feel a stand alone p-value is meaningless, and the sillyness of using a p-value threshold of 0.05 or even 0.01 for everything is not only clear with small sample sizes, but now with advanced computers and large data sets, what should be apparent anyway, is again obvious.
The newest issue of Significance magazine carries the following short article at the very beginning (page 4): "Is this the end of 'statistical significance'?" In particular it mentions an editorial in The American Statistician (where I wrote a letter in 1987 about practical concerns regarding this, not long before I stopped even trying to use hypothesis tests at all), and a "comment" in Nature. In those two cases, Significance says that it was said that researchers should stop using the term "statistical significance."
Of course you do not have to leave it at a lone p-value. You can make comparisons. You can investigate effect size. But I also see even full hypothesis tests as trying to reach a "yes/no" conclusion, when often you really want to know degree. My latest research is on saying why you should really estimate the coefficient of heteroscedasticity rather than try to say do you, or do you not have heteroscedasticity. (Further, in that case, using work by Ken Brewer I have shown you should expect heteroscedasticity naturally in regression, unless there are data quality or model specification issues.) Further, different hypothesis tests are sensitive to different aspects. This is not always clear.
Significance notes that the recent article in The American Statistician reminds us that a p-value (of 0.05) was originally intended (by Fisher, I think), as an indicator as to whether or not more research was needed, and lament the loss of that idea. But I still think even that is weak, because it is still analogous to looking at only one end of a confidence interval.
At any rate, it seems to me that more and more statisticians are finally looking at this in a more practical manner. I remember in the 1980s in particular, it seemed that big name statisticians with a theoretical leaning spent a huge amount of effort trying to find the best way to describe the null hypothesis, and I found that to be rather useless ... not that math for the fun of it is a bad idea ... just not very helpful to researchers.
Cheers - Jim
In 1991 there was a forum on The Future of Statistical Software held by the (US) National Research Council. I noted that statistical software was misleading people by only outputting p-values for hypothesis tests, and I hoped "future" statistical software would do better:
https://books.google.com/books?id=L3PshaEOnB4C&pg=PA82&lpg=PA82&dq=future+of+statistical+software+knaub&source=bl&ots=XWJm31sMFU&sig=ACfU3U2irlP_CyRqaEi91aHKyWRjdwmgFQ&hl=en&sa=X&ved=2ahUKEwi87dmxsOniAhXNVt8KHRzLB4wQ6AEwAXoECAgQAQ#v=onepage&q=future%20of%20statistical%20software%20knaub&f=false
I now think that we should just expend more effort on estimation. For example, a test for heteroscedasticity in regression does not help when it is not a matter of do we or don't we have heteroscedasticity, but "How much do we have, and how much does this impact prediction results?"
Let us try to establish more specifically what objectivist statistical inference can demonstrate with a p value. Suppose we want to see whether a given factor influences the phenomenon studied or not: for example, we want to determine if the fact of being a farmer influences a given sub-population’s probability of migrating. The question then becomes whether the differences between the estimated probabilities of migrating for farmers and the rest of the population can be explained by chance alone or whether they diverge significantly. If these probabilities prove to be different at a preset limit, for example at a 1% ‘significance level’, then we can conclude that they do diverge, since the result observed is hardly likely to have been obtained by chance. We therefore interpret these probabilities here in frequential terms, by imagining a population larger than the one observed, from which we can draw many sub-populations at random, including the one observed. If the probability of the observed sample is too low, we shall reject the tested hypothesis. This finding is consistent with our statement: objectivist statistical inference makes it possible to test the probability of obtaining the observed sample—if the hypothesis is true—but not the probability of the hypothesis itself, which is either true or false.
Why would Bayesian Analysis do anything for this issue? You still have an interval for your estimate. The P-value is essentially derived from a confidence interval....
Yes I agree that there are attempts to introduce what is called posterior predictive p-values, for example by Xiao-Li-Meng in 1984. But as he clearly said:
" For those who have called for complete dismissal of p-values (at least in the context of hyppothesis testing), such a partial Bayesian approach is certainly in the "wrong track" as it helps a "devil". This paper, however, takes the opinion that as long as "subject matter journals are flooded with p-values" ... and various ad hoc methods (eg. inserting an estimate for the nuisance parameter) are being used, there is some benefit to have an unified method for constructing motivated p-values that are always computable."
I have for the moment no more to add to this clear position on a partial Bayesian approach for journals flooded with p-values in 1984.
Have survived since 2004 without ever mentioning p-values in any of my papers. Thanks for sharing!
I am one of those researchers who had been schooled to use P values to determine whether results agrees or disagrees with the research hypothesis. Reading Wassertain et al., 2019 paper and the discussions herein, I have learnt something really important, which is quite practical and sensible. Thank you Gudeta Weldesemayat Sileshi
P values address only one question of how likely your data assume TRUE null hypothesis. It cannot evaluate if sample was unusual. It does not also measure support for alternative hypothesis. Sir Ronald Fisher who introduced p values never intended for them to be the deciding factor in a RIGID process, instead he considers them to be just one part of a process that incorporates scientific reasoning and analysis leading to scientific conclusions. I think p values can be used as long as you have full information on the data used and proper interpretation made based on clear understanding of the subject matter. However, if a Journal is against the use of p values, avoid p values in your articles to such Journals. The American Statistics Association (ASA) in 2016 reached a consensus on this subject.
After looking at this thread again, I would like to make a correction to what appears to me now to be a misleading impression I think I may have made. In my April 13, 2019 response here, this was included:
"I also see even full hypothesis tests [more than just a p-value] as trying to reach a 'yes/no' conclusion, when often you really want to know degree. My latest research is on saying why you should really estimate the coefficient of heteroscedasticity rather than try to say do you, or do you not have heteroscedasticity. (Further, in that case, using work by Ken Brewer I have shown you should expect heteroscedasticity naturally in regression, unless there are data quality or model specification issues.)"
It would have been better for me to have not said "...using work by Ken Brewer I have shown...," but instead "...demonstrating work by Ken Brewer, I have further illustrated that...."
This was Ken Brewer's idea, as noted below. I have only promoted it.
I would like to note that Ken Brewer, in Brewer, K.R.W.(2002), Combined Survey Sampling Inference: Weighing Basu's Elephants, Arnold: London and Oxford University Press, in the middle of page 111, gave an illustration as to why, for "sample survey populations," we generally have heteroscedasticity, and the degree of heteroscedasticity generally falls within a specific range, which Ken also discussed at the bottom of page 87, and noted elsewhere in his book. This degree of heteroscedasticity is related to a discussion of cluster sampling, found in Cochran, W.G.(1977), Sampling Techniques, 3rd ed, John Wiley & Sons, on page 243. There we see that variance within a cluster is related to this issue, and that is related to variance within grids in older crop studies.
I have a project, https://www.researchgate.net/project/OLS-Regression-Should-Not-Be-a-Default-for-WLS-Regression, which is with regard to this issue. It contains an update on the uselessness of hypothesis testing for heteroscedasticity. There is a paper, https://www.researchgate.net/publication/320853387_Essential_Heteroscedasticity, which explains and discusses Ken Brewer's idea. Another paper, https://www.researchgate.net/publication/333642828_Estimating_the_Coefficient_of_Heteroscedasticity, and a spreadsheet, https://www.researchgate.net/publication/333659087_Tool_for_estimating_coefficient_of_heteroscedasticityxlsx, show how to estimate the coefficient of heteroscedasticity. I presented various examples to try to cover a variety of possibilities. Methodology there also had been discussed with Ken Brewer who had suggested one of two methods there to me some years ago. The other was a shortened version of one I devised in the early 1990s.
Also, "nonessential heteroscedasticity," can either enhance or reduce the impact of heteroscedasticity, and there is a paper on that, associated with the project mentioned. Further, it is interesting to try to see in general why heteroscedasticity may sometimes appear to be absent when arguably it should be present. That can be because it is there, but not noticed, or there is a more subtle problem, or perhaps artificial data were used. One of the project updates is a short, preliminary paper on this topic.
With regard to the question here, I'll reiterate that an hypothesis test for this does you no good, and I refer you to the project linked above.
But I wrote this response because I think my April 13, 2019 response did not capture that the work for my project noted here is really just "value added" (I hope) to Ken Brewer's work, published in his 2002 book. It was Ken's idea. What I said on this in my April 13, 2019 response does not read to me now the way I meant it to read. - Thank you.
PS - Jahan just asked for the alternative to a p-value. First, a p-value is incomplete and needs information on effect size, such as a type II error probability analysis, but beyond that, estimation such as noted above, for the coefficient of heteroscedasticity, is useful. Often a confidence interval, or a prediction interval, is useful. Some complete hypothesis tests (not just p-values) may be helpful some places. - Note that both p-values and standard errors are changed by a change in sample size, but standard errors are more interpretable. A single p-value by itself is not meaningful.
As I noted about heteroscedasticity, hypothesis tests which just infer that something is or isn't true, can be very misleading. As for using them to make a decision, this can represent a shortcoming in meeting the analyst's obligation to study the true situation. Most decisions would be based not on a condition being either true or false, but on to what degree it is true, and how that impacts the situation in the application. This brings us back to the quoted second paragraph above.
(Some hypothesis tests compare simple [point] hypotheses, most comparable to a confidence interval - see "sequential hypothesis tests" - but that may not often be the case.)
Dear james
It is a pleasure, as always, to read your clear expositions on these topics that have so many angles that they can lead the analyst to get lost in different dead ends.
We must then propose alternative to p-value after seeing that all comments are against the misuse of this parameter !
The p-value is used only to determine whether to reject the null hypothesis
If an alternative to the p-value is wanted, often a standard error, if appropriate, would be better. Both are changed with a change in sample size, but the standard error is more easily interpretable. Also, a single statistic is often not adequate. In the case of a standard error, this may be remedied to a degree by comparing the standard error to the parameter itself, perhaps forming a relative standard error.
But it is often inadequate to rely on a few statistics. Graphics, such as scatterplots, often tell us much more.
For ppl interested, may I suggest the book "Reasoning with Data: An Introduction to Traditional and Bayesian Statistics Using R". It is particularly good for researchers with less solid statistical foundations, like me. It compares the traditional vs Bayesian approaches of statistical tests. It also covers the OP's concern of course.
- Badges
- Science topic
- Similar topics
- Education