If I want to test two hypothesis, i.e. whether Treatment 1 (T1) affects outcome O, and whether Treatment (T2) affects outcome O, relative to a control group (C), does it make a difference with respect to the Type I error by, whether I
a) have one control group for each treatment group, i.e. compare means of outcomes in C1 = T1, and C2 = T2
b) have only one control group, i.e. compare means of outcomes in C1 = T1, and C1 = T2,
each with a two-sample t-test (there should be no significant difference of outcomes between C1 and C2).
Does b) inflate the Type I error because I use C1 for both tests?
Thanks in advance.
It does if you use independent samples t-tests for both tests, because you are slightly violating the assumptions of independence. However, if you use one-way ANOVA and consider yourself with three groups (T1, T2, and C1), your Type 1 error is stable because your analysis correctly estimates the group structure.
Hendrik,
Multiple comparisons typically result in inflating the Type I error (unless corrected for, using various error correcting methods). In this case based on what you described, I would suggest combining the control groups towards a larger sample size for your intended statistical inference (determining whether treatment 1 or 2 is significantly influencing the Outcome/response), which in turn would decrease the Type I error probability.
If I have understood correctly, having a single control group reduces Type I error rate because the test statistics are positively correlated. Type I error correction that ignores this correlation (e.g., unmodified Bonferroni) will be overly conservative.
Power is increased by combing the control groups as Armin noted.
A "type-I error" is defined in A/B-testing (Neyman/Pearson hypothesis testing) but not in Fisherian significance testing (where the aim is to reject a null hypothesis). Unfortunately, the term "type-I error" is often used also in cases of significance testing where it is then understood as the probability to reject the null under the assumption of the null hypothesis being true. This probability does not inflate by conducting more than one test, as is always refers to a single test.
What depends on the number of tests is the "family-wise error-rate" (FWER), the probability to at wrongly reject at least one of the tested nulls under the assumption that all tested nulls are true (again adopted from Neymanian A/B-testing).
The questions are:
- is the assumption of all tested nulls being true sensible in the current context?
- does the control of the "FWER" make sense / is it desired in the current context?
- and if so, at what level should the FWER be controlled in the current context?
To my understanding, reporting the uncorrected p-values in the context is sufficient. p-values must anyway not be interpreted out of context, and when the reader is informed about this context (what includes the information on the number of tests that were performed) he/she can judge what he/she may deem "significant" and if he/she likes to follow the interpretation of the author.
- Badges
- Science topic