If I want to test two hypothesis, i.e. whether Treatment 1 (T1) affects outcome O, and whether Treatment (T2) affects outcome O, relative to a control group (C), does it make a difference with respect to the Type I error by, whether I
a) have one control group for each treatment group, i.e. compare means of outcomes in C1 = T1, and C2 = T2
b) have only one control group, i.e. compare means of outcomes in C1 = T1, and C1 = T2,
each with a two-sample t-test (there should be no significant difference of outcomes between C1 and C2).
Does b) inflate the Type I error because I use C1 for both tests?
Thanks in advance.
It does if you use independent samples t-tests for both tests, because you are slightly violating the assumptions of independence. However, if you use one-way ANOVA and consider yourself with three groups (T1, T2, and C1), your Type 1 error is stable because your analysis correctly estimates the group structure.
Hendrik,
Multiple comparisons typically result in inflating the Type I error (unless corrected for, using various error correcting methods). In this case based on what you described, I would suggest combining the control groups towards a larger sample size for your intended statistical inference (determining whether treatment 1 or 2 is significantly influencing the Outcome/response), which in turn would decrease the Type I error probability.
If I have understood correctly, having a single control group reduces Type I error rate because the test statistics are positively correlated. Type I error correction that ignores this correlation (e.g., unmodified Bonferroni) will be overly conservative.
Power is increased by combing the control groups as Armin noted.
- Badges
- Science topic