I am wondering under what circumstances is it more appropriate to use an alpha value of .01 instead of the standard .05 (using a T-test for equal or unequal variances). I have some data in which almost every group is significantly different from almost every other group when using the alpha value of .05, but not .01. I am not well educated in statistics so any help would be greatly appreciated.
Dear Debra,
Reducing the alpha level from 0.05 to 0.01 reduces the chance of a false positive (called a Type I error) but it also makes it harder to detect differences with a t-test. Any significant results you might obtain would therefore be more trustworthy but there would probably be less of them.
At our University we use informal language to describe the strength of evidence provide by a probability value in different categories:
- Probability > 0.1: No evidence
- Probability between 0.05 and 0.1: Weak evidence
- Probability between 0.01 and 0.05: Evidence
- Probability between 0.001 and 0.01: Strong evidence
- Probability < 0.001: Very strong evidence
The alpha level you set relates to the formal decision you make rather than this informal interpretation, but both can be reported together.
Lower alpha levels are sometimes used when you are carrying out multiple tests at the same time. A common approach is to divide the alpha level by the number of tests being carried out. For example, if you needed to carry out 5 tests you might set your initial alpha level at 0.05 then divide it by 5 to obtain the alpha level of 0.01. Again, the reason for this is to reduce the Type I error risk.
We also have a guide here:
https://www.researchgate.net/publication/275018715_Statistical_Hypothesis_Testing
Technical Report Statistical Hypothesis Testing
Dear Debra,
Reducing the alpha level from 0.05 to 0.01 reduces the chance of a false positive (called a Type I error) but it also makes it harder to detect differences with a t-test. Any significant results you might obtain would therefore be more trustworthy but there would probably be less of them.
At our University we use informal language to describe the strength of evidence provide by a probability value in different categories:
- Probability > 0.1: No evidence
- Probability between 0.05 and 0.1: Weak evidence
- Probability between 0.01 and 0.05: Evidence
- Probability between 0.001 and 0.01: Strong evidence
- Probability < 0.001: Very strong evidence
The alpha level you set relates to the formal decision you make rather than this informal interpretation, but both can be reported together.
Lower alpha levels are sometimes used when you are carrying out multiple tests at the same time. A common approach is to divide the alpha level by the number of tests being carried out. For example, if you needed to carry out 5 tests you might set your initial alpha level at 0.05 then divide it by 5 to obtain the alpha level of 0.01. Again, the reason for this is to reduce the Type I error risk.
We also have a guide here:
https://www.researchgate.net/publication/275018715_Statistical_Hypothesis_Testing
Technical Report Statistical Hypothesis Testing
If you are making all pairwise comparisons between several groups, please check for Tukey honestly significant difference as a way to solve your concern. Much more powerful than the method of dividing all p by the number of tests, unless you have some strong departures from the analysis of variance model.
Note that whatever the analysis you make finally, the choice of 0.05 or 0.01 as the significance level is purely arbitrary, you can choose whatever you want, as long as you explicitly give it and give p-values of your test (and not only "NS" or "p < 0.05") so that the reader may use its own one.
A couple of things. First, sole reliance on p values is a poor strategy. Second, if exploring relationships among several variables, it is important if you have hypotheses about how the variables should be related. These should guide your analyses. Thhe last resort would be things like pairwise comparisons, but if doing these consider follow Emmanuel;s advice and decide whether methods to control Type 1 error are appropriate (if so Holm's procedure seems the most recommended rather than HSD or the Bonferroni method suggested in the first reply) or whether you want methods that also take into account Type 2 error. It should not be an automatic choice to control for the familywise error rate at the expense of other things. Finally, the list of p value meaning from above (which echo lists by folks like Fisher and Efron so are meaningful), are based on single hypothesis explorations. In Efron's Large Scale Inference book (also there is a chapter in his recent Computer Age Statistical inference book) covers dealing with multiple hypotheses.
One difference between Peter's and Fisher's descriptions is that Fisher stresses that there is little difference for interpreting low values. See box 1 in https://www.researchgate.net/publication/228676951_Ten_statisticians_and_their_impacts_for_psychologists
Efron's descriptions are a few pages later.
Article Ten statisticians and their impacts for psychologists
Peter Samuels Is it appropriate to assume that the positive results from an alpha value of 0.01 are a subset of those from 0.05? Or are there cases where the 0.01 results might be different?
If you are making the exact same test, since 0.01 < 0.05, p < 0.01 ⇒ p < 0.05. If you are making different tests, everything can appear.
Hi
hope this paper helps, which appeared in a journal Abacus
Article Choosing the Level of Significance: A Decision‐theoretic Approach
I also attach my working paper, which uses my R package OptSig:
https://cran.r-project.org/web/packages/OptSig/index.html
- Badges
- Science topic