I was listening to a presentation by prominent psychotherapist/researcher awhile back and the presenter mentioned that when doing comparative effectiveness research on different components of the same treatment, the investigator needs to have a pretty large sample size to detect differences between the different components. The comment makes sense, but does anyone know of an article that discusses this issue?
Hello Glen,
As I understand the issue, the bigger the sample size the higher likelihood of 'detecting' statistically significant effect sizes. For example in t tests the mean difference of 1.5 (150 vs 151.5) may not be statistically significant when the sample size is 30 but the same difference for the sample of 150 may be significant. Research with small sample sizes 'suffers' from being underpowered often.
Regards, Witold
One thing to caution about is that "effectiveness" and statistical significance are not always the same thing. In your example, if you have an "effectiveness" of a treatment that lowers blood pressure by 5 points, it may be statistically significant with 100 participants but not with 50 participants. So, even if the effect is the same (lower BP by 5 points) it may or may not be statistically significant based on sample size. If someone asked, what is the effect of your treatment, you would answer -5 in each sample. But if someone asked if it was statistically significant, you could only say it was for the sample with 100 people. Organizations have recognized the problem with this logic and are starting to emphasize measures of effect (i.e., effect sizes) over traditional tests of statistical significance. You don't want to miss out on a potentially beneficial treatment with a large effect just because you fell a few people short of the magic p=.05.
I would recommend Cohen's (1994) The Earth is Round (p
I am not clear on what you mean by "different components of the same treatment." I lack the context to interpret this clearly.
Other cautions.
1) Just because it is statistically significant does not make it important. Consider a drug that lowers blood pressure. For every unit increase in dose, blood pressure decreases by 0.002 points. The regression is highly significant, but it accounts for only 4% of the variability in the data. This sort of problem becomes greater with large sample sizes because the large sample size enables detection of smaller effect sizes. Thus you need to figure out what effect size makes a meaningful contribution to patient outcomes.
Ideally, as a patient I would like to know probabilities for observing the intended effect and the effect size. I also want the probabilities for different side effects and probable severities. I would also like to know the conditions of the testing done to arrive at these figures. For balance I would also like the same information provided in the same format for alternative choices.
2) Be aware of Simpson's Paradox (https://en.wikipedia.org/wiki/Simpson%27s_paradox and this web site also http://optimalprediction.com/files/pdf/V2A25.pdf). In addition to the latter article, Paul Yarnold has written a number of articles on this topic that can be found here: http://odajournal.com/.
My personal preference is to make the sample size as large as I possibly can given time, budget, and ethics considerations. I would rather argue that I have a statistically significant effect that has a trivial consequence than to miss an effect because I tried to get away with a small sample size. That said, there are always exceptions.
As Ebert points out - the larger the sample size the smaller the effect for which you can show statistical significance.
If you are measuring the effect of a number of components, then you are faced with the problem of multiple testing and this requires adjustment of probability values resulting in larger sample sizes as the number of tests increase to show significant statistical effects of the same size. Bonfferroni inequality is a simple adjustment, but is conservative and other methods of adjustment for alpha levels have been published. Check the literature for problems associated with multiple testing. Adjustments for multiple testing are often overlooked even in the published literature but they are important and should not be ignored.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size