So, the way things are right now is that a significant result should be followed by considering the size of the effect; because statistical significance is one thing and practical significance is another.
But what do I do when I get a non-significant result. It seems to me that I can’t just conclude that there was an effect (e.g., d = .5) but I lacked power and therefore if I increase the sample size I will find a significant d = .5. That is, I cannot conclude that there was an effect despite the fact that there was an effect (i.e., d = .5). I can’t draw this conclusion because the because of the non-significant result which means that the resulting effect size cannot be trusted.
Perhaps the effect size I found was due to chance and a properly powered study might come up with a completely different effect size. So, increasing sample size may result in a completely different conclusion. Basically, a significant result prompts me to consider the effect size but with a non-significant results I can conclude very little. Am I understanding this correctly?
So, if recent years has thought us that a significant result is NOT the firm proof of an effect we so desire; what about a non-significant result?
The effect size depends on the variable and the treatment (or which groups) you are investigating. You don't know this value, but you may have reasonable assumptions about it or you may estimate it from some data you have observed.
This is one thing you must distinguish: The effect size is a kind of an "abstract" measure, like an "idea of a difference". It refers to some kind of model we have about the world, in which we imagine a "data generating process" with particular statistical properties that we assume. Observed data, in constrast, is something concrete, and it can contain some information about the effect size if it is put in the context of that imagined data geerating process. This is why we can use data to refine our models or even substitute them by other models to improve the mach between our models and our experience (i.e.: data).
It's a bit confusing that estimates from sample data are called the same as the abstract quantities specifying our models. An observed mean difference between groups is often called the "observed effect size", but it actually is the mean observed difference, possibly used as an estimate of an effect size.
Having said that, it should be clear that larger samples will only give you more precise estimates, as they contain more information about the effect size. Statistical significance is used to measure or value the information in the observed data w.r.t. a particular model, including a hypothesis about a particular effect size (within this model). If the data are not statistically significant, we conclude that the information is not sufficient to distinguish the estimate from the hypothesized effect size. If they are statistically significant, then there is enough information and we may claim that we should expect the (unknown) effect size being either larger or smaller than this hypothesized value. Typically, this hypothetical value is zero, in which case a "significant" result would allow us to interpret the sign of the estimate. Notably, the confidence interval is the set of all hypothetical values that we cannot distinguish statistically from the estimate.
If you plan a study, you need to hava some idea about the effect size. This idea should be reasonable, and it should not (grossly) contradict our experience. It may be the minimum effect size for which you would really want a "significant" result. It may also be an effect size that you would expect, at the best of your current knowledge. Having some previous estimates can be helpful to pin down such a value. But this previous estimate can be very imprecise - a fact you might consider, too. In your example (d = 0.5, not stat. significant) this means that at least any value between d=0 and d=1 and likely an even wider range is statistcally compatible with the observed data. Although the observed data is most likely for d=0.5, it may not be much less likely for d=0.2 or for d=0.3, or for ... d=0.8. It is still up to you what you consider a more resonable (or relevant) effect. This is something you assume, based on your expert knowedge, your experience, and your understanding. Based on this assumption you can then calculate the sample size that is likely sufficient to provide enough information about the effect size so that the estimate from this sample can be interpreted in a meaningful way.
PS: Note that significance tests are NOT done to conclude whether or not there is an effect. This is impossible. Significance tests are about the observed data (given a hypothesis about an effect). They are not about an effect. If the hypothesized effect is zero and the test is not significant, it only means that your estimate cannot be statistically distinguished from zero, so you can not even conclude the direction of the effect (the data are statistically compatible with both, positive and negative effects).
Significance tests are used to assess if a “treatment” (or sample) is significantly different from other “treatments” (or samples): we say that “there is an effect”…, with a stated risk… (if we state it)…
If one wants he can state the “intensity of the effect”, and test it with a significance test.
Data say that, with their distribution and the derived sampling distribution of the used statistics.
If you have two alternative hypotheses for your parameter [H0 and H1] to assess their significance difference you must increase your sample sizes until the Confidence interval includes one of the two and leave the other in the "complement" interval
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size