Such "guidelines" are given by the community of researchers in your field. In life-sciences it is typically 95%. This is just a value many (reviewers, editors, and readers) agree on. It is NOT because this value had any particular meaning or benefit. Actually, the vast majority of these editors, reviewer and readers calling for "significant" results and a 95% confidence, won't be able to correctly state what "significance" (in the statistical sense) means, and they won't be able to correctly interpret a confidence interval or a p-value. And, funnily, although almost noone really knows what all this is and means, everyone wants to have it, read it, show it...

There is actually one one setting where you can define a reasonable confidence limit: if you can calculate/estimate the costs of wrong decisions and the win of correct decisions. But this is a mission impossible in reserach anway! (Not so in production) So there is NO WAY to define a reasonable confidence limit in science. It would be wise to acknowledge this, but most people go the easy way and simply follow the wishes of the other editors, reviewers, and readers ....

95% CI is the minimum level to authenticate your data is statistically significant.

You can't reduce CI as per your requirement.

Good luck.

Such "guidelines" are given by the community of researchers in your field. In life-sciences it is typically 95%. This is just a value many (reviewers, editors, and readers) agree on. It is NOT because this value had any particular meaning or benefit. Actually, the vast majority of these editors, reviewer and readers calling for "significant" results and a 95% confidence, won't be able to correctly state what "significance" (in the statistical sense) means, and they won't be able to correctly interpret a confidence interval or a p-value. And, funnily, although almost noone really knows what all this is and means, everyone wants to have it, read it, show it...

There is actually one one setting where you can define a reasonable confidence limit: if you can calculate/estimate the costs of wrong decisions and the win of correct decisions. But this is a mission impossible in reserach anway! (Not so in production) So there is NO WAY to define a reasonable confidence limit in science. It would be wise to acknowledge this, but most people go the easy way and simply follow the wishes of the other editors, reviewers, and readers ....

To get around the difficulty you mention with differing expectations, a common practice is to report results for all three "commonly used" levels of significance (alpha values), namely, 1%, 5%, and 10%. I have seen this applied in several journal articles and have used the approach myself on advice from a statistician.

It based on the alpha level at which you do your statistical analysis.. If 0.05 your confidence interval will be 95%, if 0.1 your confidence interval will be 90%.

Hi,

1) p < .05 is a matter of sample size, with big sample, everything (effect size) will be significant.

2) Ziliak et al. (2008) quote the “sizeless researcher”, who is the researcher that says that something is significant (p < .05), but do not explicit the effect size (what is very common in pharmaceutical researches).

3) It is better to have p < .05, but to pay attention on the discussion of the “practical significance”.

4) In some cases p > .05 because, indeed, Ho is true (no effect, for practical matters), but it could be a problem of low statistical Power (small sample size).

I suggest to show that the Power > 0.8 when p > .05.

G*Power 3 (http://www.gpower.hhu.de/) could be very helpful.

Ziliak, S. T., & McCloskey, D. N. (2008). The Cult of Statistical Significance: how the standard error costs us jobs, justice, and lives (p. 321). Ann Arbor, MI: The University of Michigan Press.

Best regards,

Bido

Completely agree with Jochen Wilhelm. There is no absolute cut-off value. In the first place you must assess what are the “consequences” of deciding that “whatever you are studying” IS significant (palfa) when in fact IT IS (beta or type II error). The balance of the cost of these two wrong decisions should help you (I'm being optimistic...) to set a suitable alfa (and also beta value). This is a simplistic description of the problem. In practice, you need to tweak some other issues as the number of observations, your experimental design, the kind of statistical analysis, etc.

Ah, this is a nagging question for climate data. To make a long story short,

climate data is VERY noisy and the noise has temporal scales as well. In addition, the baseline has multidecadel systematic features in it. I have generally found that straight statistical averaging, etc, of climate data does not return robust results. Climate data, on a daily basis, is also not normally distributed. I have found that using the data, on a monthly basis, to produce baselines and then calculating

individual z-scores with respect to the baseline does produce some results and

some signals. Some of that method, I think, appears in my projects are on

ReserachGate - if its not there, I plan to add it soon.

The basic bottom line of what I am trying to say is that the "standard deviation" in climate data is a very nuanced statistic and its dominated by natural climate

variability. If you can figure out a way to weight climate data that helps a lot.

Statistical significance of a particular data (work) is not aimed to get your paper published in journal according to the guidelines set by the editor or referees.

Significance of your work supported by appropriate statistical analysis adds the value to the subject, and it depends on the nature and type of work performed by the researcher concerned.

When we agree on 95% CI, we allow 5% chances of error to happen (P

In case of hypothesis testing you just report the confidence level of your analysis as is (it is not restricted to "round" numbers). In case of parameter interval estimations it is really only your choice which confidence level to use. The general recommendation is to use the confidence level which is used in the most publications in your field of research because it makes much easier to compare results in this case.

Anycase define alfa error"a priori"

If you are analysing a very serious illnes without treatment you can try with p < 0.1

In the other hand a non serious illnes with a lot of possible treatments must be analysed with

p

Javier, I am scared of medical doctors who judge the quality/efficacy of a drug by the p-value. Consider I had cancer and there would be two experimental drugs available: in the published studies, drug A had p

When sample size is adequate (should be estimated a priory) confidence limit should be theory driven. In social sciences stick to the usual alpha, but in pharmaceutical research sometimes even p = .20 outcome may have theoretical implication and help the majority of the targeted morbidity! In conclusion, stats should be theory-driven and not vice-versa!

Deciding on which confidence interval to use after you have analysed your data is not a good statistical practice. The decision has to be made apriori depending on the risk you are willing to accept by accepting or rejecting the null hypothesis and the statistical power of your experiment. Some specific fields commonly use either one of 90% or 95%. Before proceeding with the excution of your experiment, you should review literature in your specific field and be familiar with the usual practice, but the altermate decision is for you to decide which one you want to use based on the intended use of your results and the risk you are willing to take.

Note that 95% confidence level takes care of type I and type II errors. That is why it is better to use a 95% confidence interval than 90%. 90% wil give you a higher probability of committing type II error.

The idea of constructing confidence interval is to trap the true population parameter, which is unknown. So changing confidence interval may not be able to give us a correct interval where the true population parameter lies.

In small studies, with a small sample size, using p < .10 can be justified because you simply may not have the power to detect a difference with significance at .05. Sometimes only one more patient with the outcome of interest is all it takes to put significance into the .05 range or out of that range. And as others have said, the effect size is also very important (the main important thing really).

But if you already "play around" with the level of significance, wouldn't it be smarter to interpret the estimated effect size directly (and forget completely about p)?

Two things are important in a study like this. One is the effect size, e.g., the difference in means between two groups. The other is the statistical significance or p-value which is attached to that difference. A difference can be very small but statistically significant. This can occur for example in blood pressure studies with many many people: a difference in 3 mm of Hg may be significant, but clinically not useful. In other studies, small studies especially, you may find a large difference between the two groups but it will not be statistically significant because of the small sample size. In that case you should note the small sample size and if possible compute the statistical power to detect a difference with the sample size you're using or have available. I have seen many small studies where the effect size is potentially clinically important, but the significance (p-value) is too large because the available sample size is too small. This can be misleading because this effect size really may be significant in larger populations.

Sometimes the best conclusion to draw in small studies like this is to state that the results are not significant with an alpha of .05, but are with an alpha of .10, and that the study does not have enough statistical power to detect a .05 difference because of the small size, but further study on a larger group is justified. The major point is that the results are not statistically significant because the power to detect this difference is just not there, not (necessarily) because there isn't a meaningful difference between the two groups. This puts the focus on the sample size, not on the difference between the two groups.

P-value and confidence intervals are related but are different things!

P value, error probablity, means how probable your conclusions in the rejection of null hipoteisis are wright...p > 0,5 means you have 5 chances (or less) on 100 to acepting the null hipotesis even it is wright...

The confidence intervel means how big is the interval variation in the means variable values can change and they still belong to the same population (or sample).

If you use a medicin to make the fever down, for example on 100 pacients, and in all of them the fever down 0,01 º C, you will have a p< 0,00000000.......

If you use another medicin on 100 pacients and in 65 the fever down 3,5 º C you wil get a P greater than the anterior, probably significant, depends on what happen on the others 35....

But the magnitudes of diferences are more interesting...

When you use 90% confidence interval you will permit considering equal more things that are diferent than when use 95%

So the answer to your question is: how big is the margin of error you are willing to accept?, As you can see, with a 90% confident interval you will have often significant statistical diferences than with a 95% confident interval...Thats your error criteria who decide!

Some of data 95% or 99%, critical data as pesticides or bioactive component, or some biological experimnet 0.01 or 0.05 and 0.001 or 0.005. Depend on sensetivity of experiment

In biomedical pilot studies, the sample sizes are usually small and the objective is to decide if further studies are needed, 90% confidence intervals or P

@Mingfu: There is nothing like "real" and "unreal" nonsignificance. Eitehr a result is significant or it is not. This is a result of a classification, typically based on some test statistic. The decision, however, is subjective and has nothing to do with the data or with "reality" and sample zies or whatsoever. The decision can be made based on expertise for the single case (->Fisher) or mechanically following a strict procedure garanteeing to not exceed an upper limit of loss in the long run (->Neyman). The problem with the latter is that losses and wins have to be quantified and compared, a cost-benefit ratio has to be determined and reasonable desired ratio has to be defined. According to these points, alpha and beta can be determined, then the sample size can be determined, and then the experiment can be conducted. I think that this is rarely the case in science anyway (I think it is usually not even possible to reasonably define/quantify losses and wins in research at all!). However, the desired long-run cost-benefit balance is again a very subjective decision or includes a series of subjective decisions about which costs and wins to consider, how to measure them and how to weight or relate them.

At the end, a *researcher* classifies a finding either as significant or not (if he/she likes to make this classification at all, instead of -better!- interpreting the effect sizes based on expertise) - neither the data nor the p-value does. When I have a small sample and decide to find the result "non-significant", why should I go on and look for a larger sample size? If, on the other side, I would consider a replication with a larger sample, then this demonstrates that I well *do* find the result significant, although the actual data might occur with a rather high probability under the null hypothesis.

This seems to be a never ending problem . It seems at first glance that my answer is not related to the question but certainly it is. This is due to the fact that if a point estimate is not within the limits of a confidence interval, we can reject Ho, and if outside, we in fact can reject Ho.

There are economic reasons that could cause differences at 5 % or 10 % to be important, for example in the weight gains of cattle. But it is also true that there are ethical and national security reasons that would required the significance levels to be 0.001 or less. The levels of statistical significance of 5 % or 1 % were established for convenience, because in the past we did not have computer development to report the true values ​​for which there are significant differences.

Let 's illustrate this concept with simple example: A man is driving a car around a curve in the road and a truck is in front of him. He stated the null hypothesis that there is not car coming in the opposite direction against the alternative hypothesis, a car is coming in the opposite. He must take a decision. And he takes the desicion to go ahead.

What if he make the type I error (rejecting Ho when it is true), the driver is died.

What if he make the type II error (not rejecting Ho when it is false), the driver is safe.

The next question is how we should stated the null hypothesis, in this sense what we have taught, is to take as the null hypothesis what is normally accepted for example the production of a crop. The alternative is usually taken as that it is smaller or as it is greataer than that value, and in the worst case as an inequality which shows little security in what the researcher is looking for.

That is a reason why we set the significance level to a value of 5 % or 1 %, and ever our desicion should be THERE IS NO EVIDENCE FOR REJECTING THE NULL HYPOTHESIS, We can only take the decision of accepting Ho, after we we have calculated the probability of error TYPE II.

Please read: Ott, R. L. (1992). An Introduction to Statistical Methods and Data Analysis. 4th edition. Pp 215-226.

If the hypothesis testing is one tail test (non inferiority studies) there is a rationale to use a 90% confidence interval.

Neyman, Fisher and Pearson are the statisticians who set these levels for significance. It is strictly considered at 5% or 1%. So I think there is no question of marginally significant by saying P < 0.1. This is what most of the serious researchers consider.

No, Sania, clearly not. Have you ever read papers and/or book from N, P and F? Neyman stated explicitely that type I AND type II error rates must be chosen based on desired cost-benefit ratios that have to be chosen wisely depending on the context. Fisher considered this as impossibel in basic reserach and was looking for a way to get a hint if *further* reaserch on a topic would be feasable. He suggested to use the p-value for this purpose but never set a strict level. He particularily stressed that the interpretation depends on the context. In his own papers he did not use a fixed, constant cut-off.

Hi Jochen,

Fine in my study I got a p value 0.051 and standard coefficient 0.056. should I say my hypothesis is supported or not. what do you mean by cost-Benefit ratios. how to take decision based on the contex If I say this is insignificant and accept null hypothesis, is it not increasing the type I error?

Hi Sania,

"should I say my hypothesis is supported or not" - Support is always relative. Ther is nothing like "it is supported or not". There is only "hypothesis A supports the data better than hypothesis B" (also note that the support is for the data and not for the hypothesis!). Therefore it is actually crucial to have a defined alternative hypothesis. Then you can see if your data is better supported by the null or by the alternative hypothesis.

"how to take decision" - when decisions are based on data alone (NP), you can not judge the quality or correctness of your decision. All that can be done is to control error rates statistically (i.e. in the long run). The type I error is to wrongly reject H0 ("false positive" or "false alarm"). When in a long run H0's are rejected whenever p < alpha, then alpha is the expected upper limit of the type I error rate.

But how to chose a reasonable value for alpha? - It depends! How reasonable is it to control the type-I error rate for an individual experiment at all? (I really don't know and I personally find it not very sensible) And if it should be controlled: at what level? "Convention" sais at 5% (alpha=0.05), but the source of this "convention" is mere confusion. NP said that one needs to quantify expected wins and losses by correct and wrong decisions to find a reasonable value for alpha. I have no idea how this should be obtained in research! If you reject H0 from such experiments when p