Hi everyone,
my question is related to post hoc when dealing with huge size effect, e.g. Cohen's d> 2.
No one asked me to perform this kind of analysis, it is just a matter of personal curiosity. It is claimed that in post hoc power analyses, p-value is in a 1-to-1 relation with the observed power, and this is clearly true. However, for huge size effects, I used a Monte Carlo simulation to understand if this holds anyway. Surprisingly, the curve shows a quite big area instead of a line (you can see it attached, N=1000 experiments from normally distributed populations).
Am I wrong to say that for huge size effects, PHP analysis gives reasonable results?
Hello Luigi,
I think the apparent incongruence comes from the fact that p-value addresses one question, whereas post hoc power estimation answers a completely different one. Thus, they won't agree, except in some trivial ways. Here's one:
If a result is found to be statistically significant (at your chosen alpha level), you may be sure that, the power to detect an effect equal or greater in magnitude to that observed in your data set was at least .50.
As well, when effect sizes are enormous (as your example of a Cohen's d of 2.0 or larger), then a priori power is going to be very high in almost every case (as will post hoc power): It doesn't take many data points to determine if there is a battleship in your back yard pool!
However, the likelihood that the observed effect size in a given sample corresponds perfectly to what you, or any other decision-maker would deem as the noteworthy, is virtually nil. (Nor that it matches the population ES, for that matter.) For this reason alone, post hoc power estimation is largely a wasted and uninformative effort (in my opinion, though others have stated this and other objections far more eloquently).
Good luck with your work!
References:
Levine M., & Ensom, M. H. H. (2001). Post hoc power analysis: an idea whose time has passed? Pharmacotherapy, 21, 405–409. doi:10.1592/phco.21.5.405.34503
Yuan, K-H, & Maxwell, S. (2005). On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics, 30, 141-167. doi: 10.3102/10769986030002141
Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55, 19-24. doi: 10.1198/000313001300339897
Dear David Morse
first of all thank you for your answer.
I have attached a pic for d=0.33 (instead of d=2.35) and some more simulated points, and, of course, you have a power of at least 0.50 for a confidence level of 0.05. So when you say "thus, they won't agree" what do you mean more specifically? From what I understood, one of the most criticized aspects of post hoc power analysis is the 1-to-1 relation between p-value and power estimated - which, in my reference case for d=2.35, is not true due to a very large scattering.
P.S. There is a typo on y-axis, the unit is dimensionless, not %.
Given power is determined by a small set of things, like effect size (so in pre-study power, the min effect size to detect) and sample size, you should think if reporting power adds anything to reporting these.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python