Hello. I am calculating the sample size needed for a study. The effect size is one input parameter for the calculation. However, there is not much relevant literature allowing me to estimate the effect size. I have planned to recruit participants in three steps- 10%, 45%, and another 45%. The main reason is that there might be significantly fewer participants in some cells.
Question 1: Can I calculate the effect size using the data I get in the first recruitment to identify an effect size for the larger-scale study?
Question 2: If so, is it better to calculate the effect size again after getting data from the second recruitment?
Question 3: To estimate the sample size needed before starting the first recruitment, what effect size would you suggest? I have found some literature, which are using different designs. I am debating between the effect size calculated based on the literature and Cohen's small effect sizes.
Thank you for your time and help!
Yes. Statisticians call such a sample a pilot sample. It's often used to learn something about the variance of a target population. Best, David Booth
Hmm, I wouldn't call it a "pilot sample". A pilot study is a feasability study to see if there are practical hurdels that one has not thought of.
A study to estimate an effect size must actually be at least as large as a study conducted with the aim to reject the hypothesis of a zero effect. Theoretically, this would make no sense at all.
However, we sometimes need to have some idea about the effect size (and the variance) we are looking for (and we have to cope with). Ideally, this idea should be derived fromtheoretical considerations about the model and theory underying the panned experiment. It's nothing more than an educated guess - we can (and must) only argue that this is reasonable in light of the theory.
Sometimes we don't have any good quantitative theory (what is not uncommon, but what is scientifically sad!), so there is not way to derive reasonable estimates from theory. Well, in this case some kind of experience is all that can help. Experience means some data. It will give you an estimation that could be used. But consider the huge uncertainty in these estimates from small samples. It's usually a good ide to take the lower limit of the confidence interval of the estimated effect and the upper limit of the confidence limit of the estimated variance to be on the save side. Its easy to see that most assumed effect sizes used to plan studies must be vasty over-estimated. This is well explainedby a blog post of Andrew Gelman:
https://statmodeling.stat.columbia.edu/2017/12/04/80-power-lie/
The problem is certainly more severe if one uses published results instead of own data (as Andrew's post is about), because there you have a publication bias (if you only look at statistically significant results, the effects will be considerably over-estimated). But I still think that using a "worse case scenario" from own data is still a good idea.
David Eugene Booth Jochen Wilhelm Thank you very much for your extremely helpful answers. If I understand them correctly, it is acceptable to use a smaller sample size to estimate an effect size for a full-scale study. The estimation is not accurate, but it can be better than an estimate based on others' published results. May I know how to calculate the sample size needed for the "screening" study? (I heard about some thumbs up rules in a pilot study. For instance, 12 subjects for one cell. I am not sure they can be applied here.)
Thanks again!
It doesn't matter the only reason for the pilot study(BTW u might wish to Google the term) is to estimate roughly the sample variance which you need for your sample size calculation. Any size that gets a reasonable estimate of variance is fine. Low cost is often the main consideration in a pilot study. Best, David Booth
Ok:
https://en.wikipedia.org/wiki/Pilot_experiment :
--> feasability study
Article The Role and Interpretation of Pilot Studies in Clinical Research
--> " The purpose of conducting a pilot study is to examine the feasibility of an approach that is intended to be used in a larger scale study. The roles and limitations of pilot studies are described here using a clinical trial as an example. A pilot study can be used to evaluate the feasibility of recruitment, randomization, retention, assessment procedures, new methods, and implementation of the novel intervention. "
Article Introduction of a pilot study
After a long paragraph on the purpose of pilot studies that resemble the points of the paper cited above, the very last point is: "... and provide estimates for sample size calculation". It's a bit misleading here and difficult to understand that provisionof these estimates is not a purpose of a pilot study. It's something that can be done (as well) with the data from a pilot study.
In Germany, if you want to apply for a study (e.g. for animal research) you must state if the study is a pilot study or a contrast study. The application is usually not approved if you say that it's apilot study and that you want to estimate effect sizes - without stating and focusing on feasability aspects. In fact, if possible problems with feasability is not likely, it's difficult to impossible to get a study approved that is done (and maybe needed) to get an impression about the effect and the variance. It is requested that these estimates are derived by theroy and analogy. You should not kill even a single animal just to get an "educated guess" of low precision. Better make some hopefully (and arguably) reasonable guess out of your stomach without sacrificing animals.
David Eugene Booth Jochen Wilhelm Thank you very much for your time and answers!
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size