Hi All,
I have conducted a pilot study with a small sample (50ppts), 2 groups - intervention (n=34) and control (n=16), pre-post-follow-up design.
I've done linear multiple regressions (similar to ANCOVAs) to see differences between groups post-test adjusted for baseline. As a secondary analysis, I performed mixed-effects models (instead of repeated measures ANOVAs due to missing data issues) to see predicted changes between groups (and post hoc contrasts within groups) over time.
Now I want to compute the effect sizes and predict optimal sample size. As far as I understand it is the G Power calculation, but I am not sure how to do it post mixed-effects models (since the manuals don't provide this option).
Could you please suggest a solution? Thanks
Hello Anastasiia,
What effect metric do you wish to determine for each analysis? What do you mean by optimal sample size...do you mean, what recommended N for a future study would result from plugging in the ES that you obtained in your pilot study? If so, be sure that you and other domain experts would agree that the observed pilot ES is a realistic characterization of what is the minimally "noteworthy" effect in the context of your study.
For mixed effects models, you're likely far better off to evaluate questions of power and sample size via simulation. Here's some links that offer just that sort of option:
1. http://www.bristol.ac.uk/cmm/software/mlpowsim/
2. libraries lme4 and lmerTest in R: https://cran.r-project.org/web/packages/lme4/lme4.pdf
3. simR in R: Article SIMR: An R package for power analysis of generalized linear ...
Good luck with your work.
Hello Anastasiia,
What effect metric do you wish to determine for each analysis? What do you mean by optimal sample size...do you mean, what recommended N for a future study would result from plugging in the ES that you obtained in your pilot study? If so, be sure that you and other domain experts would agree that the observed pilot ES is a realistic characterization of what is the minimally "noteworthy" effect in the context of your study.
For mixed effects models, you're likely far better off to evaluate questions of power and sample size via simulation. Here's some links that offer just that sort of option:
1. http://www.bristol.ac.uk/cmm/software/mlpowsim/
2. libraries lme4 and lmerTest in R: https://cran.r-project.org/web/packages/lme4/lme4.pdf
3. simR in R: Article SIMR: An R package for power analysis of generalized linear ...
Good luck with your work.
Dear Anastasiia G Kovalenko ,
It is not normal practice to carry out a sample size calculation after you have already collected data and analysed it using different methods. David Morse is correct that sample size calculations depend upon the design of your experiment, the effect size you are looking to detect and the standard level of confidence by which you wish to detect them (normally set at 0.8).
Reverse power analysis appears to be another term for retrospective or post hoc power. Generally this is deemed a bad idea. In particular using the observed effect size leads to circularity because the power calculation is essentially monotonic function of the observed p value for the test. In simple cases this boils down to the trivial observation that if you get a just significant result (e.g., p = .05) the observed power is .5 (50%) indicating that it could easily have flipped the other way.
You can do something more principled using an a prior effect size estimate, but "optimal sample size" is then simply a function of the effect size you wanted to detect (and a single effect size value is essentially not that interesting, useful). It certainly wouldn't be optimal in any convincing general sense.
More useful you could look a power curve of some kind. For example plot power as a function of a range of (a priori) plausible effect sizes for your observed n and chosen alpha level. This is much more useful but potentially still problematic. In a complicated design the level of observed collinearity, clustering within subjects, distributional issues etc. would also impact power and so you might want to include these factors in the plot. This ends up at essentially a simulation where you look at the impact of plausible values of a range of factors on power.
None of this is capable of discovering an optimal sample size without strong assumptions and constraints. For example you can factor in cost or risk of harm as well as assumptions about the data and derive something optimal for those conditions, but not optimal in general (except maybe that optimal sample size is infinity for any research question assuming zero cost or zero risk of harm to participants).
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size