Hi!
I have 4 groups to compare and I am comparing several categorical or continuous variables (total 11 variables) across these 4 groups.
The categorical variables were compared with chi square test and the continuous variables were compared with kruskal-wallis test (with the general p value, not pairwise comparisons). This means that I do not need to apply any multiple testing correction within each variable.
Now I have a question that might be very basic, but I cannot find a definitive answer.
Should I apply any kind of multiple testing correction to my results? And if so, which method is better? And what is the m (number of comparisons) that I should consider: is it the number of p values (so 11, the same number of variables) or should I multiply the number of variables by the number of groups?
Thank you!
Hello Maria,
The choice is up to you, but the reality of testing multiple hypotheses is this: The more significance tests you run, the greater the likelihood of at least one type I error (false positive difference) showing up, by chance alone.
The worst-case scenario is that the aggregate risk (when all tests are independent) has a maximum value of Prob = 1 - (1 - pc_alpha)^c, where:
Prob is the probability of committing at least one type I error in the set of tests;
pc_alpha is the per-comparison or per-test alpha level used; and
c is the number of tests to be conducted.
For you, 11 is the number of hypothesis tests you know that you will conduct (so, using the equation above, the aggregate probability, without adjustment, is ~.43. If you're opting for post hoc tests only in the case of a significant omnibus result, then you're adopting Fisher's logic as in his LSD method (as applies to the risk associated with the narrower, follow-up tests).
Good luck with your work.
Hello Maria Chiara Gelmi. Given that your discipline is ophthalmology, I wonder if your question pertains to the usual "Table 1" in many medical journal articles. I wonder further if the primary analysis is some kind of regression model that is reported in a subsequent table. If that is the context, I would say that it is quite common to not make any multiplicity adjustment in Table 1. If that is not the context for your question, please tell us what the context is. Thanks.
Hi,
The general suggestion is Bonferroni correction. This is available in all standard stat packages.
Thanks everyone for your comments!
Bruce Weaver indeed: the further analysis is a logistic regression model with the 4 groups re-organised into 2 groups as dependent variable. However, in the regression model I have only included a few of the variables that were significant in table 1, not all the significant variables. I am afraid that adding too many variables to the regression model might compromise the analysis. Moreover, some of the factors in table 1 are related to each other.
Bhogaraju Anand if I apply Bonferroni, do I have to use 11 as the number of comparisons?
You may be interested in this modelling approach whereby unreliable estimates are down-weighted in the analysis (shrinkage)
Article Uncovering interactions in multivariate contingency tables: ...
Article Using Shrinkage in Multilevel Models to Understand Intersect...
Number of comparisons 11
Alpha value 0.05
Corrected alpha=0.05/11=0.0045
Thank you! Would it be useful to perform multiple correspondence analysis to study the relationship between variables?
- Badges
- Science method