In a surprise event yesterday, the American Statistical Association (ASA), in a scientific article published on 08-03-2016, called for a halt to reliance on the value of P-value (statistical significance) in statistical studies. More precisely, researchers have called for the use of this tool and immediate cessation to use it as a semi-unique decision-making tool.
read the article The ASA's Statement on p-Values: Context, Process,
and Purpose
What is your opinion on the subject ؟.
Ali -
What this is referencing is that a p-value should not be used alone, without some kind of type II error analysis, and in particular, the practice of using a threshold of 0.05 (or 0.01, or any arbitrary threshold not based on the standard deviation and sample size, effect size, or similar), as a general practice, is problematic.
Also, the term "significance" is highly misleading.
In general, you can make better, more practical decisions when you consider standard errors which are based on standard deviations and sample sizes, and forget about going to a p-value. Even the t-statistic, like an inverse relative standard error, comparing standard error to the parameter of interest, is more meaningful.
In the past, people often might have said, and still might say, that a small, expensive, destructive sample for an industrial application would prove a specification had been met, when all it really proved was that the sample size was too small to prove that it did not meet specifications when you set a threshold as low as 0.05. Now, with large databases available (besides the problem of representativeness) we see that a threshold of 0.05 is often obviously, and ridiculously, high. This is because it was never a good idea to have an arbitrary threshold at all. Generally, population standard deviation and sample size are straightforwardly important.
Sometimes an hypothesis test might be used when relative standard errors or perhaps confidence intervals (which also require distribution knowledge unless you are sure that the central limit theorem comes into use, or you use Chebyshev's inequality) are not very helpful. But you need to do a sensitivity analysis. Consider this: https://www.researchgate.net/publication/271072864_Analyzing_n_Samples_of_2_Observations_Each
I think statisticians may sometimes tend to blame researchers using statistics, for misusing p-values, but I long ago thought that too many statisticians gave the wrong impression by expending so much academic effort on studying p-values, when a practical approach does not stop with a p-value. At least, I don't think that was clearly communicated for a long time. Consider the following letter to The American Statistician, in 1987:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
Anyway, in my opinion, p-values have always been far overrated, as a lone p-value is virtually meaningless.
And the word "significance" in statistics has long been greatly misleading. It was a misnomer from the beginning.
This was supposed to just tell someone when research should continue or not, but using an arbitrary threshold was always a bad idea, even for that purpose, in my opinion.
[One more note: historically, as I recall (but I have not thought of this very much in a very long time), there were always two rival schools of thought - and I think that one did not include type II error analyses. But you really need that.*]
Cheers - Jim
* FYI: I guess that puts me with Neyman here, I think, though maybe not when it comes to using models in survey statistics, which I support and he possibly might not have, even if using randomization. :-)
I took a look at the article titled ''The ASA's Statement on p-Values: Context, Process, and Purpose'' (https://amstat.tandfonline.com/doi/pdf/10.1080/00031305.2016.1154108?needAccess=true). Following for the moment.
Ali -
What this is referencing is that a p-value should not be used alone, without some kind of type II error analysis, and in particular, the practice of using a threshold of 0.05 (or 0.01, or any arbitrary threshold not based on the standard deviation and sample size, effect size, or similar), as a general practice, is problematic.
Also, the term "significance" is highly misleading.
In general, you can make better, more practical decisions when you consider standard errors which are based on standard deviations and sample sizes, and forget about going to a p-value. Even the t-statistic, like an inverse relative standard error, comparing standard error to the parameter of interest, is more meaningful.
In the past, people often might have said, and still might say, that a small, expensive, destructive sample for an industrial application would prove a specification had been met, when all it really proved was that the sample size was too small to prove that it did not meet specifications when you set a threshold as low as 0.05. Now, with large databases available (besides the problem of representativeness) we see that a threshold of 0.05 is often obviously, and ridiculously, high. This is because it was never a good idea to have an arbitrary threshold at all. Generally, population standard deviation and sample size are straightforwardly important.
Sometimes an hypothesis test might be used when relative standard errors or perhaps confidence intervals (which also require distribution knowledge unless you are sure that the central limit theorem comes into use, or you use Chebyshev's inequality) are not very helpful. But you need to do a sensitivity analysis. Consider this: https://www.researchgate.net/publication/271072864_Analyzing_n_Samples_of_2_Observations_Each
I think statisticians may sometimes tend to blame researchers using statistics, for misusing p-values, but I long ago thought that too many statisticians gave the wrong impression by expending so much academic effort on studying p-values, when a practical approach does not stop with a p-value. At least, I don't think that was clearly communicated for a long time. Consider the following letter to The American Statistician, in 1987:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
Anyway, in my opinion, p-values have always been far overrated, as a lone p-value is virtually meaningless.
And the word "significance" in statistics has long been greatly misleading. It was a misnomer from the beginning.
This was supposed to just tell someone when research should continue or not, but using an arbitrary threshold was always a bad idea, even for that purpose, in my opinion.
[One more note: historically, as I recall (but I have not thought of this very much in a very long time), there were always two rival schools of thought - and I think that one did not include type II error analyses. But you really need that.*]
Cheers - Jim
* FYI: I guess that puts me with Neyman here, I think, though maybe not when it comes to using models in survey statistics, which I support and he possibly might not have, even if using randomization. :-)
The word "significance" is fuzzy. It is subjective. How significant is this significance? To me it appears to be right to restrict only certain % of samples deviating from the mean behavior ( whatever the domain be) of the population. It is better to prescribe the threshold rather than p value. It is rather prudent to say xx percentile of the sample satisfied the fitness/evaluation criterion.
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude