Hi,
I am planning a follow-up experiment to a previous study and have a question about estimating sample size:
For the sake of simplicity, the previous study showed that Condition 1 had more accurate responses (64%) than Condition 2 (41%) and a paired samples t-test was significant (t(12)=3.43,p=0.005, d=0.9, two-tailed). This finding has recently been challenged because the stimuli are somewhat confounded. My new experiment investigates whether a significant effect will be observed between condition 1 and condition 2 when new, more appropriate stimuli (after removing the confound) are used.
The problem is that when the new stimuli are used, the difference between conditions will *probably* be much smaller because we're supposedly removing a confound (e.g., ~10% rather than 23% difference), although we still hypothesise it will be significant. To calculate the sample size needed for a significant effect with the new stimuli, I will need to estimate cohen's d but it's almost impossible to estimate this precisely because I don't know how much the confound contributed to the initial effect. So, what cohen's d should I use for my power calculation? I see 3 options:
(i) I simply predict what cohen's d may be by reducing the original finding (e.g., from 0.9 to 0.4)?
(ii) specify a minimally desired cohen's d that would satisfy the psychological reality of the effect (e.g., based on d=0.3, power=0.8, p=0.05, the estimate is 89 participants)? But what if, after collecting the data my d is slightly below this yet still meaningful (e.g., d=0.25)?
(iii) collect preliminary data (I've sometimes seen this suggested but it seems counter-intuitive to the whole pre-planning idea).
Many thanks for any insights!
Ryan
Ryan - I'm just saying: what if the "you" in your question, is aiming for the same Cohen's d, say, a moderate 0.4 - even after more appropriate stimuli are considered (and after removing the confound). See how to put all this into equation (and what else changes), for an accurate new sample size calculation.
P.S. If you use the same statistical test(s), then go with the standard idea of finding an equilibrium between: significance level, power level, effect size and sample size.
IMO, either of the first two approaches is fine. Just be clear which approach you used when your report it. As for the second part of option ii, that's just a risk one has to live with.
approach 2 is more sensible, approach 1 is still acceptable and approach 3 kicks the can down the road and misses the point of power calculations. Power calculations are based on ensuring that a minimally useful difference is likely to be detected with an acceptable level of risk, i.e. it reduces the risk of meaningful effects being missed.
The issue you raise with approach two is a smokescreen. Power calculations and the resulting statistical analysis are inherently bound to each other (which is why retrospective power analysis is not useful, it is a restatement of the results in another way). Thus if you power based on what you consider to be a minimally useful Cohen d, carry out the experiment and analysis to the letter and obtain a value below the minimal value then it will by mathematical definition fail to achieve your pre-defined alpha level. If you feel a Cohen's d of 0.25 would be useful then you should use that in your powering. You can't move the goal post afterwards.
What is minimally useful? Depends on your application and the benefits and risks associated with it.
These articles may provide guidance for proposing a Bayesian inference in experimental studies that use student t's to quantify evidence for study hypotheses
Article Bayes factors for superiority, non-inferiority, and equivale...
Article Bayesian alternatives to null hypothesis significance testin...
https://link.springer.com/content/pdf/10.1186/s12874-020-00968-2.pdf
Ryan - I'm just saying: what if the "you" in your question, is aiming for the same Cohen's d, say, a moderate 0.4 - even after more appropriate stimuli are considered (and after removing the confound). See how to put all this into equation (and what else changes), for an accurate new sample size calculation.
P.S. If you use the same statistical test(s), then go with the standard idea of finding an equilibrium between: significance level, power level, effect size and sample size.
، لكن تظل الدراسات الحديثة المعاصرة الأكثر صدقا في التجارب، والمقارنات
تتأثر الدراسة أو النظرية بما قبلها وبعدها سلباً وإيجاباً
- Badges
- Science topic
- Similar topics
- Psychology