Hi everyone.
I am running an ANOVA analysis, and there is an interaction with a medium partial eta squared value (.06). Still, the p-value is insignificant ( .11). I have calculated the sample size based on previous studies, and it is ok for my experiment. So, what should I consider? How should I understand this data? Thanks.
Hello Livia,
The brief answer is, the nonsignificant result means that the target effect can not be claimed to be different from zero.
If you conducted a thoughtful, a priori, power analysis and determined that your sample was sufficient to detect differences of at least your declared target effect size (however quantified), with an acceptable level of power, then the nonsigificant result implies that population differences do not reach the magnitude of your target ES.
However, if your sample size determination was based more on what other studies have used, rather than on detecting a variance-accounted-for (whether partial or full model) of six percent, then your nonsignificant result could have been due to: (a) too little power to detect such an effect (and, hence, insufficient N); or (b) a correct failure to reject Ho (the target ES does not exist in your target population).
Good luck with your work.
If you have a method to calculate the confidence interval for the effect size, that is often instructive. Likely you would find the confidence interval for the effect size crosses 0.
Just focus on your effect size. How does it compare to other studies and is it what you expected? P-values are not very helpful - see the ASA statement: ‘… it is time to stop using the term “statistically significant” entirely. Nor should variants such as “significantly different,” “p < 0.05,” and “nonsignificant” survive, whether expressed in words, by asterisks in a table, or in some other way.’
(Wasserstein et al., 2019, Moving to a World Beyond “p
Let me add that the small, medium, and large labels applied to various effect sizes are quite arbitrary. IMO, using them as the basis for sample size estimation is not advisable. Here are some relevant resources.
https://homepage.stat.uiowa.edu/~rlenth/Power/2badHabits.pdf
Article Avoid Cohen’s ‘Small’, ‘Medium’, and ‘Large’ for Power Analysis
André I Wierdsma , the problem with simply looking at the effect size statistic is that, particularly with low sample sizes, the effect size can be relatively large, but there may no reason to believe that the effect size of the sampled population is actually larger than, say, zero.
I would argue that it's fine to present effect size statistics with confidence intervals, and ignore p-values. But that comes to essentially the same thing as looking at the p-value.
Consider a simple case
A = (1, 3)
B = (2, 4)
Considering these two groups, Cohen's d is about 0.7.
Or thinking about the linear model B = A + e, r-squared is 1.
But in neither case would we have any confidence that these effects are meaningful.
Sal Mangiafico , the questions was: "How should I understand this data?". Surely it's all about combining substantive and statistical ideas. We evaluate the effect size in the context of the research question, results of previous studies, and research characteristics - including sample size.
- Badges
- Science topic
- Similar topics
- Biological Science
- Botany
- Pharmaceutical Botany
- Plant Extracts