Hi again,
We conducted a study which was consisted of 6 repeated measures performed on a single group.We used repeated measures one-way ANOVA for that purpose, and received significant results, with an effect size of approximately 0.35.
Since we did not performed sample size calculation beforehand, we were asked by the reviewer to perform a post-hoc power analysis.
As far as I know (I might be totally wrong) - power analysis aims to verify that we avoid type II error, which is: there is a difference in the population, but we missed it (due to too small sample size, for example).
So, my question is - if we did find significance in our results, there is actually no chance that we will have type II error. This stands in line with the fact that power analysis calculators returns a power of 1.00. So- what's the point of doing it?
Thanks,
Uri.
You are right, there is really no point in doing a post-hoc power analysis. As Maarten said, a post-hoc power analysis never makes any sense. You may cite the work of Neyman and Pearson to substatiate your reply to the reviewer.
A totally different point of view is a look at and a thorough interpretation of a confidence interval (CI). If gives you the range of effect sizes that are "not too incompatible" with your data. You may check if the CI if it excludes irrelevantly small effect sizes (if so, you could confidently claim that the data would not support irrelevantly small effect sizes - otherwise, despite the "significant" result, you may not claim this!) or if it extends to relevant effect sizes (if so, you could say that your data is compatible with assuming relevant effect sizes - otherwise, despite the "significant" result, you may not claim this!). Ideally, the CI is completely with a range of relevant effect sizes. In the worst case, the CI is completely with the range of irrelevant effect sizes. If it is only partially extending into relevant effect sizes, the interpretation is less confident (like I said above: the data is not too incompatible with relevant effect sizes, but also not with irrelevant effect sizes...).
If you look at and interpret the CI, the "significance" becomes far less relevant. The only point is that for any "non-significant" result the CI will include 0 (or the "no-effect") and thus definitely extend into "non-relevant" effect sizes. Having a "significant" results only sais that you may claim the sign of the effect, but it tells you nothing about if the data is (in-)compatible with (ir-)relevant effect sizes. This can only be judged by looking at the CI.
The difficult part is, for sure, to say what a relevant effect may be. If you have no good arguments or a sensible opinion(!) about that, there is no point in interpreting a CI, as there is no point in doing an ad-hoc power analysis (as this also requires to state the size of a relevant effect that should be detected with a specified power).
Hi Uri,
there is really never a good reason to do a post-hoc power analysis. There is much written about this but this formulates it nicely: http://daniellakens.blogspot.com/2014/12/observed-power-and-what-to-do-if-your.html
You are right, there is really no point in doing a post-hoc power analysis. As Maarten said, a post-hoc power analysis never makes any sense. You may cite the work of Neyman and Pearson to substatiate your reply to the reviewer.
A totally different point of view is a look at and a thorough interpretation of a confidence interval (CI). If gives you the range of effect sizes that are "not too incompatible" with your data. You may check if the CI if it excludes irrelevantly small effect sizes (if so, you could confidently claim that the data would not support irrelevantly small effect sizes - otherwise, despite the "significant" result, you may not claim this!) or if it extends to relevant effect sizes (if so, you could say that your data is compatible with assuming relevant effect sizes - otherwise, despite the "significant" result, you may not claim this!). Ideally, the CI is completely with a range of relevant effect sizes. In the worst case, the CI is completely with the range of irrelevant effect sizes. If it is only partially extending into relevant effect sizes, the interpretation is less confident (like I said above: the data is not too incompatible with relevant effect sizes, but also not with irrelevant effect sizes...).
If you look at and interpret the CI, the "significance" becomes far less relevant. The only point is that for any "non-significant" result the CI will include 0 (or the "no-effect") and thus definitely extend into "non-relevant" effect sizes. Having a "significant" results only sais that you may claim the sign of the effect, but it tells you nothing about if the data is (in-)compatible with (ir-)relevant effect sizes. This can only be judged by looking at the CI.
The difficult part is, for sure, to say what a relevant effect may be. If you have no good arguments or a sensible opinion(!) about that, there is no point in interpreting a CI, as there is no point in doing an ad-hoc power analysis (as this also requires to state the size of a relevant effect that should be detected with a specified power).
Here's another short note that describes the main problems with post-hoc power analysis (as it is typically done).
- http://homepage.stat.uiowa.edu/~rlenth/Power/2badHabits.pdf
Jochen Wilhelm Please can you send some link or further reading on how to interpret this confidence interval and significant effect size relevance in case we do get non-significant results? appreciate any info and help
Thanks
E.g.: J. Neyman and E. S. Pearson (1933) On the Problem of the Most Efficient Tests of Statistical Hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character Vol. 231: 289-337
Article Outline of a theory of statistical estimation based on the c...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size