There is no authoritative reference for using 0.05 as significance level. Au contraire, there are references from Neyman as well as from Fisher that the level of significance has to be chosen based on the whole context (scientific, economic, aims, limitations ...). In the Neymanian philosophy a "conventional level" makes no sense because the fixed constraint is the cost/benefit ratio of the research, and there is anyway no possibility to sensibly decide on an acceptable level a posteriori. In the Fisherian philosophy there is no cost/benefit ratio, and the criteria to select a level are made not so explicit; the judgment is neccessarily a researcher's judgement, what always includes a major part of "personal opinion". Note that both philosophies use the "decision" for different purposes, in a different "frame of action". Neyman wants answers about hypotheses: what hypothesis can be accepted under optimum cost/benefit conditions. Fisher uses data to get a rough impression about the "significance" of a finding. If the data is too likely under the null hypothesis, it may not be worth/possible/feasable to further investigate it.

So you can cite both, Neyman and Fisher (depending on the tests you do: null-hypothesis tests or significance tests). Both argue that there can not be a "standard" or conventional level. Although there is the citation from Fisher

“If P is between 0.1 and 0.9 there is certainly no reason to suspect the hypothesis tested. If it is below 0.02 it is strongly indicated that the hypothesis fails to account for the whole of the facts. We shall not often be astray if we draw a conventional line at 0.05. . . .” (Fisher RA. Statistical methods for research workers. London: Oliver and Boyd; 1950. p. 80)

But this is one sentence, taken out of the context, besides many other statements of Fisher pointing to more flexibility. See here for a collection: http://www.jerrydallal.com/lhsp/p05.htm

i would almost always recommend to report the real p-value. why not report a p-value of - for example - 0.093? so every reader is able to see the significance or non-significance. also i recommend to calculate and report effect sizes. because you know, significance is not the full story (everything will be significant with a large enough sample size). and if you don't want to report the real p-values you may follow this convention: * p

Yes. There's the massive amount of literature on the problems and/or uselessness of significance testing (see e.g., attached link to "402 Citations Questioning the Indiscriminate Use ofNull Hypothesis Significance Tests in Observational Studies"). Fisher's approach has been thoroughly questioned in its entirety since it originated, and there remains little defense for experiments using the "hypothesis testing" paradigm. See attached papers for a small sample of critiques.

http://warnercnr.colostate.edu/~anderson/thompson1.html

I agree with Manfred: the best option is to report exact p-values that you obtained from your data. If you described your problem in more details (what your null hypothesis is and what criterion you use), then one would give you more specific answer. 

Hi, 

I agree with the comments stated before. But besides literature I would report not only the exact p values but also the effect size and even the statistical power. 

For that, you can use the R package pwr or the freeware G*power. 

Yo could also argue in your text that p-values are sensitive to sample size when the sample is not biased. 

Hope this helps!

If I were you, I would present the work as a pilot study and put emphasis on effect sizes rather than p-values. Do a power analysis, maybe, to support your assertion that sample size is too small to detect an effect.

Keep in mind that results are unlikely to be generalizeable, anyway, so I suggest you to collect more data anyway. 

(Andrew, thank you for the link. It's glorious!)

I think my problem is more simple. I am aware of the discussions about the flawness of the use of statistical significance but anyway, this is a convention and I am not interested in going into these debates in my paper. I only need to cite one or two credible sources which state that using statistical significance at the 10% can also be worth attention. 

Of course, I am reporting all statistical information - coefficients, S.E., statistical power, sample size, etc. 

Another note: the paper is in the phase "revise and resubmit" so it won't make sense to do anything more than what the reviewers ask me to do. 

Plamen - this is what you probably were looking for...

http://stats.stackexchange.com/questions/55691/regarding-p-values-why-1-and-5-why-not-6-or-10

Subhash, I need something like this, but I could not cite a forum discussion in my paper. It must be a peer reviewed paper or a book. 

An internet search may help Plamen. I know of one reference I read ten or so years back, but am unable to recall - age is at times a nuisance!

Well ... here is another one if it is of any help

http://www.nature.com/news/scientific-method-statistical-errors-1.14700

There is no authoritative reference for using 0.05 as significance level. Au contraire, there are references from Neyman as well as from Fisher that the level of significance has to be chosen based on the whole context (scientific, economic, aims, limitations ...). In the Neymanian philosophy a "conventional level" makes no sense because the fixed constraint is the cost/benefit ratio of the research, and there is anyway no possibility to sensibly decide on an acceptable level a posteriori. In the Fisherian philosophy there is no cost/benefit ratio, and the criteria to select a level are made not so explicit; the judgment is neccessarily a researcher's judgement, what always includes a major part of "personal opinion". Note that both philosophies use the "decision" for different purposes, in a different "frame of action". Neyman wants answers about hypotheses: what hypothesis can be accepted under optimum cost/benefit conditions. Fisher uses data to get a rough impression about the "significance" of a finding. If the data is too likely under the null hypothesis, it may not be worth/possible/feasable to further investigate it.

So you can cite both, Neyman and Fisher (depending on the tests you do: null-hypothesis tests or significance tests). Both argue that there can not be a "standard" or conventional level. Although there is the citation from Fisher

“If P is between 0.1 and 0.9 there is certainly no reason to suspect the hypothesis tested. If it is below 0.02 it is strongly indicated that the hypothesis fails to account for the whole of the facts. We shall not often be astray if we draw a conventional line at 0.05. . . .” (Fisher RA. Statistical methods for research workers. London: Oliver and Boyd; 1950. p. 80)

But this is one sentence, taken out of the context, besides many other statements of Fisher pointing to more flexibility. See here for a collection: http://www.jerrydallal.com/lhsp/p05.htm

I agree with Manfred.  Report the p-value, effect size, confidence interval, r-squared or whatever makes sense...  The alpha = 0.05 rule isn't magic.  In a preliminary study, it may make sense to use alpha = 0.10.  In a medical study, where people's lives are on the line, you may need a much smaller p-value...  I would try to write the results as "In a preliminary study, we find that p < 0.10 is suggestive of a significant effect that warrants further study.", or something like that.

This isn't the strongest statement, but in this quote, Zar does mention that the alpha=0.05 rule is arbitrary.  Zar is the bible for statistics in biology/agriculture.  Who is the equivalent in social sciences?

'By experience, and by convention, an [alpha] of 0.05 is typically considered to be a "small enough chance" of committing a Type I error while not being so small as to result in "too large a chance"of a Type II error (sometimes considered to be around 20%). But the 0.05 level of significance is not sacrosanct. It is an arbitrary, albeit customary, threshold for concluding that there is significant evidence against a null hypothesis. And caution should be exercised in emphatically rejecting a null hypothesis if p = 0.049 and not rejecting if p = 0.051, for in such borderline cases further examination -and perhaps repetition-of the experiment would be recommended.'

Zar, Biostatistical Analysis, 5th., p.79

 If you can get your hands on Agresti and Findlay, Statistical Methods for the Social Sciences, I would see what they have to say.  From reviews,  it looks like they endorse models and confidence intervals rather than sticking to the alpha=0.05 rule.  But I don't have the book to see.

Obtaining a sample size that is appropriate in both regards is critical for many reasons. Most importantly, a large sample size is more representative of the population, limiting the influence of outliers or extreme observations. A sufficiently large sample size is also necessary to produce results among variables that are significantly different. However, for qualitative studies, where the goal is to “reduce the chances of discovery failure,” a large sample size broadens the range of possible data and forms a better picture for analysis.

On the other hand, the degree of variability of some experimental unit imposes a threshold of variation of the variables measured in an experiment. For example, if that threshold is high, then 10% of statistical significance is enough to justify the significant differences of an used treatment compared with the control, although the size of the sample is small. Now, it is important to repeat the experiment because systematizing experiments will allow us to know that this happening is certain or reinforcement under the conditions of the experiment.

Sincerely,

Angel

There is nothing magical about the famous 0.05/5% However I cannot stress how important it is, that one understands what a p-value of 0.05 means: One in twenty times, you will get a significant results by pure chance! I.e. The result does NOT harbour anything biologically meaningful, it simply occurs by 'luck'.

In other words a p-value of 0.10 means that if you are 'lucky' and 'hit' the one in ten times a significant result occurs by pure chance... This is what your result is... A stroke of 'luck'!

Dear Leon, 

you are completely right! You know it goes the other way around though - in reality there might be a phenomenon which "by chance" does not appear very robust in the empirical analysis with the non-perfect data we always use. 

Leon, except that you can say that about any results.  You'll end up sounding like the tobacco companies in the 1950's.  "Scientists have a mountain of evidence linking cancer to cigarette smoking?  Ha!  Just meaningless correlation!"...  In this now very theoretical conversation, far from what the OP was originally looking for, I think it's important to keep a couple of things in mind:  1) the idea of science-based evidence; and 2) how evidence is evaluated by science.

1) We should be looking at experimental results in a science-based way, not just in an  evidence-based way.  Of course if you conduct a thousand correlations on random sets of data, you will get all kinds of spurious correlations.  (Results of such an exercise can be seen here: http://www.tylervigen.com/ ).  Likewise, if the sellers of woo-woo conduct a bunch of experiments on homeopathy, they will find some significant results.  But as far as we understand the world, homeopathy actually working is pretty much impossible.  We have "evidence" that homeopathy may be effective, but no one who understands either chemistry or medicine will be swayed by a few spurious positive results.

2)  Science doesn't see one p-value and declare that the result is clearly true.  Instead we conduct preliminary studies, based on our understanding of previous science.  If things look promising, we might get funding to do a more in-depth study.  If that's positive, some others might try the same or related experiments.  We use a web of evidence and a web of understanding to evaluate what think is true about the world...  True, our p-values might just be lucky, or our experiments might have been poorly done, or we might have missed the true cause of what we're seeing, but that's why we proceed this way.

I'm not writing all this to be pedantic.  If we don't follow an initial "interesting" result (p-value = 0.10, r-square = not-great, etc.),  we might  miss something potentially valuable.

But also, there are different considerations in different disciplines.  If we are trying to grow nice turfgrass, 90% certainty about some treatment might be acceptable.  If we are considering scrapping chemotherapy for some other treatment for cancer, we will want to approach the evidence differently.

And there are different standards for different places you might publish.

Steve Cherry has some interesting thoughts on this issue in the attached paper.

http://warnercnr.colostate.edu/~anderson/PDF_files/Statistical.pdf

In my experience, a finding based on even 0.05 significance doesn't hold up very well. Add an explanatory variable, apply a log transform to one or more variables, change a criterion for inclusion/exclusion a bit, and the finding evaporates. So, perhaps you need to ask why you want to publish such a tenuous finding and carefully look at other options - getting more data, discussing the "findings" informally, using the result of an analogous precursor study to establish a prior and then applying a Bayesian approahc,  etc.

I did this blog post related to p-values and 'luck': https://leonjessen.wordpress.com/2014/11/13/why-do-we-need-to-be-careful-when-performing-multiple-tests/

A generally accepted significance level is 0.05. To the library! The Web of Knowledge is a non-exhaustive electronic database of published articles. In 2014 it lists 4,131,272 articles published. Some of these will lack any statistical test, but by the same token some of these will have many statistical tests. If we assume that each article has exactly one statistical test, then a little over 200,000 articles report findings that are simply chance alone (Type I errors). Of course as Salvatore pointed out, some of this is mitigated by having multiple studies from multiple labs doing similar projects. If we move towards 0.1 as the critical value then we need to ask ourselves how many spurious conclusions are we happy reading?

One could argue that I have taken too many liberties in the above example. So in my research I have a system where I have calculated 84 variables. I have 4 treatments, so I will run 84 Tukey HSD tests to look for significant differences. With alpha=0.05, I find one variable that shows significant differences between treatments. With alpha=0.10 I find three. In either case would you be comfortable with an article that talked about significant treatment effects? I argue that this is a good case for saying that there are no treatment effects (I have 20 replicates/treatment -- not a large number, but it takes about a half day of work to get one replicate in one treatment.).

Whatever alpha you choose, I will be unimpressed by a citation. I want to know what alpha you chose, and why. Give me some indication that you understand the consequences of your choice. If there are other approaches to the analysis that could avoid the problem (say linear regression rather than multiple comparison procedures), give some indication of why they were not used. What does one gain if one goes from 0.05 to 0.10 to 0.15? Why 0.1? Is this substituting one magic value for another? I agree with others that one should report p-values, but there is still a determination that "I will discuss these differences as important," and this usually involves some significance test.

Timothy, your last paragraph is great! Well said.

But I also want to put my finger on your statement that many articles "report findings that are simply chance alone (Type I errors).":

First I think that it is a common misconception to say that a finding "is (due to) chance (alone)". Chance is a measure of our expectation of something, and no supernatural property of a result or data etc. The data is the data is the data. We may explore/calculate how likely we will expect such data given some hypotheses about the data generating process or the distribution of the data. This expectation is called "chance" and it is quantitaively measured as probability. The p-value is the probability to get a "more extreme test statistc" assuming a particular probability model about the data and the null-hypothesis about the test statistic. The words "chance" and "probability" are almost synonyms, except that "probability" adds a formal measure, whereas "chance" does not.

Second I am not sure if one can talk about "type-I errors" in this setting. Most research articles have not specified any beta and they do not seem to follow the decision-theoretic approach of Neyman. But then there are no sensible error-rates defined. What is rejected is just the "point-null hypothesis", i.e. the observed data had a too low chance to occur given the effect was exactly zero. But this is generally not a sensible (research) hypothesis. If something is done, if something is systematically different, if there is some intervention, why should we expect an exact zero effect at all? In this formal "kind-of-hypothesis-test" setting I would say that the type-I error rate is close to 0, no matter what the authors publish and at what level they test. The more important question is if the size is relevant and, at least, if the direction of the effect is correct. But this is not answered by claiming a "significant result".

I would be greatful to read other opinions and objections about this.

Jochen,

I agree with the first full paragraph. I am not sure about the second. I'll walk my way through, and try not to jump to the end before I have finished with the beginning.

"no sensible error rates define" I agree, in that the error rate used is just the standard 0.05, and this is seldom justified. It is simply accepted. So this defines, by default, the Type I error rate: If the null hypothesis is true and I get an observed value that is only seen once in 20 tries, then I will reject the null hypothesis just because I want to. The probability of identifying a false positive is 0.05. I choose not to argue if this is sensible, as I have other battles where I think I have a greater chance of winning. The other problem is that I know how to solve this problem, but I never have enough data and I have no expectation of ever being allowed to gather enough data. This is the "I have an experiment where I test a question" followed by "I repeated the entire experiment 100 times to evaluate the performance of the experimental design." However, this is a great exercise in data analysis using computer generated data.

My understanding is that the "straw man" null hypothesis is one of the main criticisms put forth by proponents of Bayesian methods. I see the value in such an approach, but I have yet to be convinced that a straw prior is a great improvement over a straw null. That said, I can also see research questions where prior data can be used to change our expectation of new results and a very good prior can be formulated.

"I would say that the type-I error rate is close to 0" is based on the idea that most data analysis is using tests where the null hypothesis is that the difference is exactly zero in an experimental design implemented for creating differences. The problem is that in my research I think that "zero" is a statement of probability. It is better to say that the null hypothesis is that the difference is indistinguishable from 0 given background variability, rather than it equals zero. Given that uncertainty, I can make a type I error. One can further point out that any p-value that we calculate is also an estimate. So if an experiment is done 10 times I would expect that I could get a p-value of 0.0069 the first time, 0.0073 the next time, and so forth. We never put 95% confidence intervals about our estimated p-values.

All that said, I will agree that it is unreasonable to expect that just because I choose to reject the null if I get a p-value less than 0.05, that I will get an actual type I error rate of exactly 0.05. My intuition says that it will be closer to 0.05 than to zero, but I have no proof that this is true.

Timothy, thank you for contributing.

I think you got my 2nd paragraph wrong, at least partly wrong. The key point is put by you: "If the null hypothesis is true [...] The probability of identifying a false positive is 0.05." - exactly this is it. The probability in this framework is defined as the limiting frequency of falsely rejecting H0, and this makes sense only if the tests are repeatedly done on true H0s. If some H0 are actually true, the actual rate of false rejections will neccesarily be lower. That's fine, alpha gives just the upper boundary of the rate. But when almost all H0s are false on a priori grounds then I do no see any point in a formal test. You can not falsely reject any H0. And a non-signifcant result is then almost certainly a false negative result.

This is interesting: "It is better to say that the null hypothesis is that the difference is indistinguishable from 0 given background variability" but this is not what hypothesis tests do. A correctly performed hypothesis test neccesarily requires to state a "minimum relevant effect" and a "minimum accepted power" (or type-II error rate, beta). Then everything starts making some sense. But without having specified the relevant effect and without having planned the study to obtain just the desired power, the absolute value of p (and that of alpha!) are not well defined. If we did the experiment as good as we coud and we get a "large" p-value, this can only be ued as hint to better stop further investigations on this topic - at least with the current experimental design. A "low" p-value indicates that either further research can be useful or it is worth interpreting the effect size in the scientific context. But the p value can not be used to control any error-rate an any sensible way. I am NOT discussing if 5% is sensible (you may substitute it by any other value). It is the *rate* itself that cannot be defined, and even if - controlling such rates won't make any sense in the this context.

Regarding your last paragraph: The type-I error rate is the (limiting) rate of false rejections (what is possible ONLY when H0 is true). So when H0 is true in ALL the tests, this rate approaches alpha (0.05 or whatever value). The lower the proportion of tests on true H0s, the lower will be the actual type-I error rate. The chosen alpha (e.g. 0.05) still and always gives the upper limit of the (accepted, long-run) rate of type-I errors. So alpha may be always and constantly 0.05, but the actual type-I error rate depends on the proportion of true H0 among the tests that are actually performed. Simple extreme case: if H0 is always always false, it is impossible to do any false rejection (any rejection is correct because H0 is always false). This demonstrates that the actual rate is zero, independent of alpha.

Thus said, you would be a bad researcher if your actual type-I error rate is close to 0.05, because this would mean that almost all of your H0s are true...

This is the researcher's presective. It gets worst when we look from the literature's perspective. Here we no not see what all was tested and how often it was tested. Here we only see more or less "selected results", most of them being "significant". If we look at these selected results only, then, among those, the rate of false positives can easily and considerably exceed alpha. If we imagine that all reasearches work always only on true H0s, and onyl the "significant" results get published, then all of the published results are false-positives (the actual type-I error rate within published results is close to 100%) ...

Jochen,

Thank you for the examples, they helped.

three type of level of significance 1%,5% and 10%. level of significance depend upon nature of study. i am student of social science study. for social sciences always taken level of significance 0.005 mean 5%.

for 1% level can i accpet 0.13 p-value or it is out of the range

No, it's out of range. For a 1% level you would reject H0 only when p

With small sample sizes, the precision of your study is low. A p-value combines two important measures – the effect size and the precision with which it has been measured – in a way that loses our ability to think about them separately. A p-value of 0.05, corresponding to t value of 2 (roughly) can be from a tiny effect measured with high precision (large sample, low variation between observations) or can be a large effect measured from a low precision study (small sample, lots of variation between observations). You cannot tell.

I would forget p-values and hairsplitting over significance, and be honest: report the effect sizes and their confidence intervals. That way you are reporting what you found out and the margin of uncertainty around your findings.

Confidence intervals not tests....agree with Conroy

But what is the effect size?

The usual maximum likelihood estimate (MLE) is only a "large-sample estimate". For small samples, the MLE can be far off any reasonable value (and that applies to a CI around the MLE as well). If there is so little information from the data, the context (not to say the prior knowledge) becomes relevant. So just for small samples, relying on the MLE and the CI may not be satisfactory. It is just the core of the testing philosophy that the MLE (nor the CI) is *not* (neccesarily) a useful indication of the effect size, and a conclusion is based on properties of the procedure rather than on the estimate (the poor-man's solution, if one is unable to put it into a proper context, that is, to specify a prior).

I think, although it is often okayish, we don't do us a favour when we unreflectedly interpret MLEs as posterior modes (or means) and CI as HPDIs (or credible intervals).

Statistical inference is based on the idea that it is possible to generalize results from a sample to the population. How can we assure that relations observed in a sample are not simply due to chance?. Significance tests are designed to offer an objective measure to inform decisions about the validity of the generalization. However, the amount of information varies wildly from problem to problem. Still the most traditional statistics is anchored in fixed significance levels, and constant tables of evidence to judge p-values. According to Pérez and Pericchi (2014), there is no clear justification to the use of fixed significance, except by tradition. On the contrary, there is a vast literature implicitly critical of it, most of it publish outside statistical journals. They recommended put forward an adaptive alpha which changes with the amount of sample information. This calibration may be interpreted as a Bayes/non-Bayes compromise, and leads to statistical consistency. The calibration can also be used to produce confidence intervals whose size take in consideration the amount of observed information.

I have enclosed the reference about it.

Pérez, María-Eglée and Pericchi, Luis Raúl (2014). Changing Statistical Significance with the Amount of Information: The Adaptive α Significance Level.

Stat Probab Lett. 85: 20–24.

sorry i meant for 10% not 1%

and what are the best tests i can use to test GOF with Tobit

Please, What do you want to say with GOF with Tobit?

i wan to define the best model using GOF tests such as R2 Chi-square, p-value

Please, What is GOF test?

GOF = Goodness Of Fit

Colleagues and I have a previously published study using a p value of 0.10 attempting to predict success in a high stakes certifying exam. We developed a predictive formula for people who were at risk of failing, and we wanted to include those who had a possibility of failing, assuming additional instruction to those predicted to pass (at the low end of the exam score) would not harm those students, while failing to add instruction for those at moderate risk might result in their failure. As Dr. Wilhelm notes, above, there is not source mandating p of 0.05 or less. I would, however, include your reason for a p value of 0.10 in your methods section. See Haas, Rule and Nugent, October 2004 Journal of Nursing Education 43(10):440-446

The choice of any level of statistical significance is arbitrary as it is not determined through optimizing an objective function.

Convention suggests the use of 1%, 5% and (less frequently) 10% levels.

Please stop promoting wrong layman's explanations.

In reality, the effect will be either positive or negative. It may be extremely close to zero, but looking precisely enough, it will still be either on the positive or on the negative side. When we reject H0: "the effect is 0", we claim that we do have enough data to "see" if it is positive or negative. If we don't reject, we say that our data is not conclusive (could be positive or negative, we don't dare to say). Thus, only when we reject we can make a claim that may be wrong.

Now consider the "true" effect is so tiny (and/or the noise is so huge, the sample size is so tiny) that we are practically completely inable to "see" it. Among tests under this scenario, we would reject about 5% of the H0s, claiming that we would see the direction (positive or negative). In fact, we must expect that then 50% of the rejected H0s will indicate a "positive" effect and 50% a "negatve" effect". Thus, we must expect that we are right 50% of the time, not 5%. So when there is no power, the test is not better than mere guessing.

The highest possible risk of being wrong is 50%, and that is independent of the level of significance at which you reject H0! The level of significance only controls the (highest possible) risk of ending up with "no conclusion" (= 1- the level of significance).

In the other extreme, if the power is close to 100% (the effect is large, the noise tiny, the sample size large), you will reject H0 with certainty, in any test. I am not absolutely sure about that but I would think that the probability of a wrong conclusion in this case is almost zero. So you will actually never make a wrong decision.

Thus, taken the two extreme scenarios together, your risk that a conclusion is wrong is somewhere between 50% (guessing) and 0% (perfect data). And interestingly, these limits do not depend on the significance level, but on the power (which is usually not known to us)!