Significance is based on your per-specified cut-off value. By convention people use 0.05, but for a pilot study or other special case, you can use 0.10 or other value too. In addition to p value, 95% CI and magnitude of OR (effect size) may be more important.

Strictly speaking, p = .053 is not significant at the alpha = .05 level. You can see that the 95% confidence interval for the odds ratio includes 1 (= no relationship). If you had a few more cases (=more power), the result may become "statistically significant" at the .05 level.

As you wrote "One of the independent variables..." I assume you tested several variables. In that case you must consider the multiple testing problem. If this is what you did, then your p-value is clearly not significant.

Christian Geiser : yes, more samples may always lead to a smaller p-value, and if there is some (possibly irrelevantly small) effect, a sufficiently large sample will certainly give a p-value < 0.05. But if the true effect is small, it may require a huge sample size. I just made a simultaion for a survival analysis with 800 patients. The p-value for the interesting coefficient in a Cox-PH model was 0.07, and the reaserchers asked how many more patients would be required to get this p below 0.05. Assuming that the new patients have (on average) just the same properties as the patients in the given sample, I calculated that data from about 1000 more patients would be required for a power of just 80%. This shows that having a p close to 0.05 does not mean that "a few more data" will make it.

And an additional note on "increasing the sample size when p

Christian Geiser , you state "strictly speaking" .053 is not < .050. Can you clarify when you mean???

Jochen Wilhelm That's interesting about the simulation. I would not have guessed that it takes that many more participants.

Regarding the second part of my response ("If you had a few more cases..."), I probably should have left this part out. I agree with you that it would not be a good idea to simply add cases until you find p < .05, and that's not what I wanted to suggest.

What I was really trying to get at with that comment was the arbitrary nature of p

Daniel Wright With "strictly speaking" I guess I was trying to indirectly express my opinion that we should not take the p values associated with null hypotheses significance tests overly serious (although I'm not saying they are irrelevant). As we all know, for most statistical tests, the validity of the p value depends on assumptions. At least in my field (psychology), many of these assumptions tend to be at least partially violated (already the rather basic the assumption of a simple random sample is typically not met in psychological research). So how much of a real difference does it make to find, for example, p = .053 vs. p = .047, given that neither of those p values is likely to be perfectly accurate in practice anyway?

Also, if you used a software that happened to give only two decimals, the value of .053 would be rounded to .05 which would be "significant." Taken together, my point is that I personally am not taking the p values overly serious. Other information (e.g., effect sizes) is more relevant in my opinion.

Dear Muhammad Badarnee, in my opnion, if the variable makes theoretical sense for your model and its inclusion improves the model's performance, I would include it. In this case, you should include an explanation in your paper to justify this decision, as well as to suggest that other authors pay attention to this point.

Another possibility is: if you have not yet applied the Stepwise technique, its application could also make your decision easier.

This is not significant at p-value=0.05

Christian Geiser , thank you for your response. I fully agree that 0.05 is arbitrary and any similar cut-off is similarily good (or bad). You mentioned that assumptions are not (never) met perfectly in practice and that we are usuayll not aware of the relevance of the deviations- therefore a "precise" (and arbitrary) cut-off is not very meaningful, anyway. I fully agree here. I just can't help to give my point of view that 0.05 is a very liberal cut-off that already accounts for all this. This is the really worst acceptable level of confidence for the interpretation of an effect (relative to the tested hypothesis). It's already not nice (but barely sufficient) to be close below this (like 0.03). There should not be any intention to extend this even above 0.05. So my point is: 0.053 is not as good as 0.047 - it is as bad as 0.047. By convention, the author of the study with p=0.047 was lucky, and the author of the (equally uninformative!) study with p=0.053 was unlucky. But that's like going (too) fast into a radar trap. There is a tolerance, but there is also a hard (and arbitrary) cut-off, and if you have to pay a fee or loose your driving licencedoes not depend on the actual endagering but only on the fact whether or not the measured speed is above or below this cut-off.

And I have another comment on your statement "Other information (e.g., effect sizes) is more relevant in my opinion.". Again I very much agree to this important point. However, this is sometimes seen as a possibility to interpret a point estimate and ignoring the uncertainty associated with it, what is not ok. For instance, if the authors find a point estimate of a difference of 9 with a 95%CI from -1 to 19 (p = 0.06), then it is not ok to argue that the effect size is about 9, because there is really very little information about the effect. The data are compatible with a large range of effect sizes, including even negative values. The p>0.05 indicates that the data are not sufficient to conclude the sign of the effect size (with acceptable confidence), so how can one dare to belive that it is "about 9"? It is, however, absolutely ok to look at the entire CI and conclude that the effect size is somewhere between -1 and +19. And if a relevant effect would be larger than 25, say, then one can conclude that the effect size is irrelevant.

The interpretation of effect sizes is important when one wants to see if the effects may be relevant. This requires to give a cut-off of a minimal relevant effect size. One can then see if the data are compatible with only irrelevant effect sizes or only relevant effect sizes, or with both. If they are compatible with only irrelevant effect sizes, they may or may not be significant* - but this does not matter. And if they are compatible only with relevant effect sizes, they are clearly significant (but usually with p

Excellent answer Jochen Wilhelm ! The last sentence in the answer is the answer.

"This means: if you interpret relevance, significance does not matter."

You cannot conduct a meaningful investigation if you do not know what is relevant. Statistical significance is a chosen data quality level. It is the minimum information level that indicates the possible sign of an effect. Statistical significance is not sufficient to claim an effect size as the uncertainty is larger than the effect. What can be claimed is that the number of data points sufficient to indicate the sign of the effect. The difference in data quality between p = 0.053 and p = 0.047 says nothing about the expectation of obtaining p < 0.05 on a repeat of the experiment.

A relevant effect size cannot be claimed at the customary level of significance.

Nicely explained by Jochen Wilhelm .

Jochen Wilhelm you are always on point and I generally agree with your perspective on significance vs estimates. We already know about publication bias, pvalue/BF hacking and cherry-picked methods, but I do not think that the burden of proof lies on the researcher.

Nowadays an epidemiologist has access to samples sizes of 300k patients, i.e. everything is significant and they can publish massively, these datasets are usually instantly available and supported by bad practices as post-hoc power analyses.

On the other hand, a field biologist needs to spend a lot of time getting samples, and because of the small sample size, the results might or might not be "significant" (i.e. p

Stefano Nembrini , I certainly agree with the very importaoint you make: that scientific rigour is most important. And also that difficult-to-obtain data from small studies is valuable and should be published. And I really mean the data, so that it can be used by others, so that others can add data and so that the data can be analysed in the context of more sources of data (see the FAIR principles).This would be a different culture of publication, where it is acknowledged and honored to plan and conduct a well-designed study and to get high-quality and reliable data including all thinkable controls with huge efforts and present it the scientific community in a FAIR way. Rejecting some statistical hypotheses (and making the corresponding claims) should not be required (although this is nice to have, for sure). I still don't think that it is ok to argue in a paper that some effect is, say, positive, because the sample estimate was positive, but the uncertainty is huge because of the small sample and the large variance. The data are very valuable, yet. But not because one could make a conclusion, but because one can add these data to the community. To achieve this, editors and reviewers must learn to value the usability and relevance of the data, independend of the question whether or not some specific research question can be clarified with these data alone. What do you think?

Stefano Nembrini I agree with a lot of what you wrote. In particular, I think that there is value in small-sample studies (if larger samples aren't feasible for acceptable reasons), whether their results are "statistically significant" or not. Many small studies on the same topic can be very valuable later on when properly aggregated through meta-analytic techniques and should therefore all be reported/published unless they have fundamental flaws in their methodology etc.

However, I also think that there is value in large-sample studies even when they report small effects (e.g., the OR of 1.15 that you mentioned as an example). Of course, the "statistical significance" of such effects may be irrelevant/misleading. Nonetheless, these studies help us understand which effects are present (i.e., unlikely to be zero) but small. That is, because of the high precision (low SE) of the estimate, we can be fairly certain that the effects are in fact small. It can be as important for us to know which effects are small (and perhaps negligible or not of practical relevance) as it is to know which effects are large/practically significant. Also, some effect sizes such as ORs may appear small but the effect can still be important when the process in question affects millions of people (e.g., how many more people die when not treated with Drug X).

Christian Geiser you definitely have a point.

What I was actually referring to - and this is will be an unpopular opinion - is mainly the average publication is more basic in terms of research question/statistics/implications, e.g. it is not uncommon to read a paper where where 1.15 could be something like the OR of access to the ER for Latinos in a certain area compared to Caucasians (or whatever), with no real discussion just a sentence that says something very generic about being more conscious that these people are more likely to use the ER, for instance.

Honestly not that often researchers deal with the life of millions of people (thank God).

Most times a researcher that tries to publish proper work (with a Pvalue larger than 0.05) is less likely to be published than someone that for whatever reason decides to perform a non parametric test, or a non parametric anova until the computer produces a P smaller than 0.05 even though neither them nor the reviewers (or even the editors) understand what they are actually did.

I constantly see people always looking at nonparametric tests as a master key that allows publication (if the MW does not work try the sign test or whatever you can find) - as if there was an actual alternative to parametric tests (like a one-to-one relationship). Like "my data is not normal", and then you tell them to plot the variable and it's clearly a mixture of two variables.

There are so many ways to generate pvalue nowadays (bootstrap, permutation, bayesian methods, asymptotics, nonparametrics), that one can literally cherry-pick the smallest, but there is so much confusion about the Pvalue, that it is often thought that they can tell you if an effect is actually present! While some other completely ignore their meaning, and are just interested in a way to increase their chances of publishing.

I have seen people so adamant that 0.053 is not significant while their paper that had a pvalue of 0.49 was the best thing ever because "they don't make the rules, the threshold is 0.05"

Very often I find people applying nonparametric tests (whose assumptions and conclusions are obscure to them), when they could be using a glm and model their outcome properly.

Why do people do that? I understand undergrads but I have seen professors do that, rather than ask a statistician.

I once suggested an engineer (and professor) to report the P value rather than writing "P

Stefano Nembrini , ad non-parametric tests:

The last five manuscripts submitted from collaborating groups were reviewed by statisticians and the all came back with the very same stereotypic response that a test of normality would not work on such small samples and that therefore non-parametric tests had to be used. This is such a shame! And I was not successful to convince the authors to argue against that. They increased the sample size, they don't mention that this is now a two-step procedure (peeking data) that actually invalidates the p-values, and they test hypotheses they don't understand. And they get it published, adding five more papers in relatively high-impact journals showing others that "such data ought to be analyzed using non-parametric tests". This is such a shame!

When a test is performed at 5% level of significance (Alpha-value=0.05), it means that there are 5% chances that a true null hypothesis would be rejected. Therefore, a test is said to be significant if the null hypothesis is rejected at the 5% level.

Dear Muhammad Badarnee, since you have set 5% (Alpha-value=0.05) as your risk level in your work. Then a P-value of 0.053 indicates a non-significant outcome.

Jochen Wilhelm Stefano Nembrini The problem is funding and publishing. These two form a circular dependence that shows 'money is the route of all evil.' That said, it is the way statistic is taught that is a prime enabler. All hail the null hypothesis.

Statisicians believe they are the embodiment of Popper's falsifiability. Create a null and test it for rejection. Let us count the ways to reject. (Oh! Right! Every statistician winks and says you really cannot accept or reject.) If you really are a Poperian let's do null hypothesis significance testing. That is real acceptance and rejection. (Well, not really, wink.)

There is no need for statistics when you have a small sample size. All you have are data. Plot it. Tabulate it. Talk about it. Most important, present it. If no one can see your data (the raw stuff) and your methods, then no one needs to believe the statistics.

Jochen Wilhelm wrote

"The last five manuscripts submitted from collaborating groups were reviewed by statisticians and the all came back with the very same stereotypic response that a test of normality would not work on such small samples and that therefore non-parametric tests had to be used."

Statistics, as taught and practiced, is a shame, Jochen.

To conclude whether the coefficient/ variable is significant or not, we consider values of the significance level. The value significance level may mostly vary from 1% to 10%. Based on the values of significance level (0.01, 0.05, 0.1, etc), we can say that the coefficient/ variable is significant if the p-value is less than the given significance level. Based on this truth, we may conclude as follow.

1) p-value of 0.053 is significant if the given significance level is 10% (0.10).

2) p-value of 0.053 is not significant if the given significance level is 1% and 5% (0.01 and 0.05, respectively).

Thank you!!!

Jochen Wilhelm without even mentioning that sequential tests (i.e. testing for normality before applying a t-test) have been shown to increase type I error, and the t-test has been shown to be robust to deviations (which also applies to an OLS).

It is such a shame that even statisticians reinforce that, and this is kind of embarrassing, and the fact that not even you were able to convince them makes me realize the issue is more intricate than I thought.

I have seen the same issues in Europe as well as the US, so at least we are consistent geographically. See the comment right before mine, it exactly sums up the whole point of "it's significant/nonsignificant", well even a pvalue of 0.7 is significant if alpha is set to 0.8, like I have a brain but it's on strike..

There are a lot of factors to consider here, which are well described in the excellent discussion above.

If we initially take out the p-value and just consider your regression variables you can get a good understanding for the integrity of your model. The coefficients themselves are a form of effect size, thus it would be worth to look up similar analyses and get a feel for how large of an effect you really have. We can also do that by considering your confidence intervals. Although these don't cross the zero threshold it does come pretty close and is a fairly wide threshold (at least without knowing if 4 is considered a large beta in this context). Assuming that b = 4.71 would actually be a pretty substantial coefficient then what this is showing you is that there is a reasonable degree of error in your model that could be due to sample size or something else. Without even considering the p-value we can get an impression that the model is ok but not necessarily giving us a rigorous interpretation of the data. Throwing the p-value back in this simply tells us about the probability that you have a genuine effect. The 5% cutoff is arbitrary but in any case your model exceeds that, and is thus not significant. You could suggest that it is a "trending effect", and this would not be an invalid interpretation. However, given what we took away from the beta and the CI you don't have a strong case on your hands as it currently stands (which is ok, a negative or approaching negative finding is still critical story telling for the scientific audience and is helpful in guiding future research). Thank you for posting this and sparking such a wonderful discussion.

Nik Murphy , I don't subscribe to this:

"You could suggest that it [the non-significant result] is a "trending effect", and this would not be an invalid interpretation."

This is okay durnig ongoing work, to fell motivated by some incomplete or low-quality data to gain hope that one is on the right track, But it is entirely meaningless when it comes to publishing.

The statistical significance is the very first line of defence against wishful thinking. It ist just that, to NOT start believing that there may be a substantial effect. Say that a value of 4 would be considered a relevant effect, and say the estimate is 5 with a confidence interval (CI) from just about -1 to 11 (so p>0.05). This just means that your data is (almost) as compatible with irrelevant effects as with relevant efects. Given the data, there is no reason to believe that there is something great is going on. If the CI was from 0 to 10, p would be just 0.05 - and still there is nothing really different. The data are still as compativle with irrelevant effects as with relevant effects. At least now there is sufficient information in the data to exclude a negative effect with acceptable confidence, but this may or may not be interesting, knowing that a relevant effect should be at least 4.

I still do not understand why someone needs a "strong case", as if a researcher had the burden of proving the null(-ified) hypothesis is false (we already know that in nature that hypothesis is always false).

We are not making a case on a new drug that we want to put on the market.

Data is data, this is what Muhammad Badarnee observed and as such it has relevance, if he had found a p-value of 0.049 the message would also be the same. It would be helpful to know the sample size.

Also, in a logistic regression, I would compute marginal effects, since the parameters are difficult to interpret.

Nik Murphy "trending p-values" is an atrocity: there is a ton of publications on the topic. https://academic.oup.com/bja/article/115/3/337/312358

Just to be consistent with what we mentioned earlier: Pvalues could be computed using permutation or the bootstrap approaches (Wild/residuals etc), you could compare the results, and this would cast some more light on the data, e.g. if asymptotics are appropriate etc. Once you have the data, might as well make good use of it, don't you think? :)

Just to bring Fisher into the mix: a pvalue of 0.053 might indicate that another study (possibly with more samples) should be performed. After all, we gain insight of a phenomenon when replicating the same experiment we get similar results, a small pvalue in one single study might as well be a fluke.

NO. Maybe you need to increase the "n". And probably will be significant.

Erick Valencia , if you mean collect more data and add this to your existing data set, check your p, and if still greater than 5% repeat, then this is wrong; your p value for this hypothesis is used. If you mean go and do a new study but with more data, and that you will have some probability for finding p less than 5% then this is true (I would say probably without knowing more details). Please clarify what you mean since someone reading your answer could think you mean the first interpretation.

I agree with Dereje in the two statements as follows:-

1) p-value of 0.053 is significant if the given significance level is 10% (0.10).

2) p-value of 0.053 is not significant if the given significance level is 1% and 5% (0.01 and 0.05, respectively

HI Muhammad Badarnee ,

I guess the p-value of 0.053 is the two-tailed result, I agree with professors that it is a significant level at p-value < 0.1 or an error at 10%. However, you may check for reliability and validity of the variables, if any item could be deleted, and Cronbach's Alpha and Factor Loading are over 0.7, the p-value could be accepted at p-value < 0.05.

First of all, why did you produce a logistic regression?

Hello Muhammad Badarnee

This value depends on the given significance level.

1) p-value of 0.053 is significant if the given significance level is 10% (0.10).

2) p-value of 0.053 is not significant if the given significance level is 1% and 5% (0.01 and 0.05, respectively).

number 2 is usually more common in research.

You must select (should have selected) the p-value you designate as significant and stick to it. My heart goes to so many researchers who have not been taught that life (hence research) is a gamble and you cannot change your bet after the dice have been rolled. If you get your pre-selected amount of less you have found something. If you do not then "a lack of evidence is not evidence of a lack" * If you have students, please warn them.

Sorry, I misplaced the source of this quote.

If u work with variance-based software. 0.5 and below is significant only.

And if u use variance-based software, 0.7 or lower is significant.