Hi,

I am afraid you will not find a very straightforward and universally valid answer. You may want to take a look at the paper attached. I transcribe the Abstract below:

"Recent advances in Bayesian hypothesis testing have led to the

development of uniformly most powerful Bayesian tests, which

represent an objective, default class of Bayesian hypothesis tests

that have the same rejection regions as classical significance tests.

Based on the correspondence between these two classes of tests,

it is possible to equate the size of classical hypothesis tests with

evidence thresholds in Bayesian tests, and to equate P values with

Bayes factors. An examination of these connections suggest that

recent concerns over the lack of reproducibility of scientific studies

can be attributed largely to the conduct of significance tests at

unjustifiably high levels of significance. To correct this problem,

evidence thresholds required for the declaration of a significant

finding should be increased to 25–50:1, and to 100–200:1 for the

declaration of a highly significant finding. In terms of classical

hypothesis tests, these evidence standards mandate the conduct

of tests at the 0.005 or 0.001 level of significance."

Best wishes

There are discussions on appropriate alpha for ecological monitoring, but I"m not aware of any for classic experimentation.

http://www.jstor.org/discover/10.2307/1942031?sid=21105507882773&uid=3737536&uid=4&uid=2

However, I think that the problem you describe is perhaps a misunderstanding about what the significance level tells you. An alpha of 0.05 essentially means that if you conclude a real effect, 5 times out of 100 you would be wrong. This is a type-I error, nothing more. It does not tell you if the effect that you observe is biologically relevant. And so, an experiment with a large sample size may have the resolution to detect very small statistically significant effects, but it is up to you to build an argument for their relevance or lack there of. 

To my knowledge, any guidelines for statistical significance thresholds (alpha) are misleading in the framework of frequentist inference.

No statistical testing would tell you the truth.

A data analyst should also keep in mind that typically he wouldn't like to reject some test (e.g. Bartlet o Levene test for homogeneity of variances), while he strongly would like to reject the null hypothesis of other test (e.g. an ANOVA or two sample T- test). The "preferred" decision depends both on the research question and on the statistical techniques used, which determine the null and alternative hypothesis. So, sometimes we wish very small p-values, and we are happier as they get smaller; other times we wish large p-values.

After rejecting a null hypothesis, one should assess the practical relevance (e.g. clinical relevance) of his result. Many tests are statistically significant but practically irrelevant. Only many independent (different samples at different places at different time) tests on the same research question can contribute to a corpus of evidence that could be studied via meta-analysis.

Moreover, when setting a significance level, one should consider how many test he is going to perform, if these test are dependent and so on. Some contribution from Alessio Farcomeni could help in this discussion:

1. A review of modern multiple hypothesis testing, with particular attention to the false discovery proportion. Scandinavian Journal of Statistics 34 (2), 275-297

2. Some results on the control of the false discovery rate under dependence. Scandinavian Journal of Statistics 34 (2)

Anyway smaller thresholds are usefull to control the false discovery rate and when reporting p-values we should use a scientific notation: if p=0.0000056, we should report p=5.6E-06.

Ye-Sle Shin -

The problem is that a single p-value is a function of sample size and is not very meaningful on its own. A power analysis or other sensitivity analysis is needed in each case. Some perform power analyses to pick a test, but what I mean here is to basically compare, for the experiment made, two hypotheses, to see which is relatively more preferable, such that the null and alternative hypotheses are analogous to the two ends of a confidence interval.

Sequential hypotheses do this by testing until you decide between two simple hypotheses.

Whenever you can use a confidence interval instead, however, that is generally much more practical, interpretable, and useful.

Setting a level of 0.5 is actually a bad habit, arrived at collectively. It really should be changed based on your sample size, as you indicated. A quick answer to your question would be that a power analysis could be used to adjust it, based on your sample size each time.

The letter at the attached link discusses this problem.

Cheers - Jim 

Article Practical Interpretation of Hypothesis Tests - letter to the...