For a correlational study, I conducted an a priori power analysis based on a previous experiment with similar measures. Given this I was able to estimate my sample size, and went using that.
However, when submitting the paper, a reviewer mentioned that the sample was too sample for yielding stable estimates and cited Lakens & Evers, (2014).
I read recently (while not well explained) that you can also conduct/report a post hoc sensitivity power analysis to corroborate a significant finding in a correlation analysis. (using gpower)
Is this correct? Can someone elaborate on how to report this in APA format, and what I can actually say. The analysis does show that my r value is higher than the critical threshold for 80% power at .05.
Thanks in advance.
Hello Mircea,
Did you mean to say that the reviewer suggested that your sample size was too small? If so, the appropriate response is that your method section should have outlined the a priori power analysis you ran, and use that as justification for the sample size (assuming that your target effect size--ES--was chosen with a suitable rationale).
Post hoc power analysis (that is, asking the question, "For the effect size I observed in my data set, how powerful would such a study be with the same sample size, same alpha level, same number of variables, to detect such an effect size in the population, should it exist?") is only useful to the extent that you consider the observed ES to be a suitable target ES. But, your a priori analysis should have addressed that point in the first place--what ES is a suitable target in the context of the study you executed. As well, the observed ES is a random variable, and highly unlikely to coincide with the population ES.
For more emphatic dismissals of post hoc power analysis, take a look at these articles:
Hoenig, J. M., & Heisey, D. M. (2012). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistican, 55, 19-24. doi:10.1198/000313001300339897
Levine, M., & Ensom, M. H. H. (2012). Post hoc power analysis: An idea whose time has passed? Pharmacotherapy, 21, 405-409. doi:
10.1592/phco.21.5.405.34503
O'Keefe, D. J. (2007). Brief report: Post hoc power, observed power, a priori power, retrospective power, prospective power, achieved power: Sorting out appropriate uses of statistical power analysis. Communication Methods and Measures, 1, 291-299. doi:10.1080/19312450701641375
Good luck with your revision!
Mircea,
I agree with David that you have some grounds to defend your methods based on calculating sample size a priori, you can even reference table 2 of the article the reviewer suggested in this defense. However, the reviewer is likely wanting you, at the very least, to provide some measure of uncertainty around your correlation estimate (e.g., 95% CI). The Lakens & Evers (2014) article relies quite heavily on Schönbrodt & Peruginib (2013), so it'd be good to look over too. Taking it a step further you can report a "corridor of stability," see the section with the same title in Lakens & Evers (2014). This basically gets at,
"An important question when designing an experiment is how many observations are needed to observe relatively stable effect size estimates, such that the effect size estimate will not change considerably when more participants are collected" (Lakens & Evers, 2014).
Also, table 1 in that article gives a nice summary relating two common effect sizes, r and d, to points of stability and power.
Best with the revision.
Cheers,
Caleb
Data Sailing From the Seas of Chaos Into the Corridor of Stability
Article At what sample size do correlations stabilize?
Hi Mircea,
Here's an online APA guide. Hopefully, it's helpful:
https://owl.english.purdue.edu/owl/section/2/10/
--Adrian
thank you Caleb A. Aldridge and David Morse ! that was exactly my impression of the situation, however, I wanted to make sure I was not overlooking something or misinterpreted something.
Very useful comments and reading! Thanks again.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size