If a trial was a single centre trial with a small number of people, and they found that a hypothetical intervention had a significant effect compared to the control group, but in-fact with a larger powered trial they would have found no significant difference. Is this described as inaccurate results or invalid results?
If the criterion for statistical significance was set at (say) 0.05, this means that there is a 5% chance of falsely rejecting the null hypothesis. If in a larger study, the difference is not statistically significant, then the most likely scenario is that in the first study, you did in fact make a Type 1 error. Type 1 errors are most likely caused by some type of bias, such as selection bias.
Small sample size can increase both alpha and beta errors. However, it usually tends to suppress the association. In your small sample size study, u found significant association (type 1error), which may be due to selection or info bias (as commneted by Adrian). However, your question was if that was ''invalid'' or inaccurate results? invalid is a synonym for inaccurate (but not a synonym for unreliable), and the common term used is ''invalid results''.
I think it is neither inaccurate nor invalid; I think that "inadequate" may be more appropriate.
A small sample size reduces the statistical power to detect a difference between intervention and control groups. Type II errors are very common with small sample sizes, although your study reports a Type I error. However increasing the sample size reduces the likelihood of both Type I and Type II errors.
Given the small sample size and that the results do not reflect the reality in the population, the results would be said to be 'inconclusive'.
Dear Peter,
Finding a significant result with a small sample size means that there was a large effect. Such a result is no less important than a similar significant result with a larger sample. The result with a small sample size should only be overlooked if other results have a much lower significance value. Alternatively, you could use a Bayesian technique to combine all the evidence together.
With a small sample size it is easier to take a sample that shows a large effect. This outcome is just sampling error. The probability of collecting an unusual sample decreases as sample size increases. As sample size increases it becomes more likely that another scientist who reruns your experiment will arrive at the same conclusions and might even see the same magnitude of effect that you observed.
Finding a significant result with a small sample size means that you were lucky. You can now publish your results and (possibly) mislead hundreds of readers. All of your conclusions are predicated on the assumption that your sample size of 5 is typical of the underlying distribution. However, this is almost never true for one sample. It is true that 1,000,000 samples of size 5 will (on average) have the same mean as the underlying population, but that does not help a lone sample of size 5.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size