Example Scenario: 1 categorical variable (Yes/No), 3 continuous dependent variables
- 3 independent sample's t tests are conducted (testing for group differences on three variables); let's assume 2 of the 3 variables are significant with medium-large effect sizes
- a binomial logistic regression is conducted for the significant predictors (for classification purposes and predictor strength via conversion of unstandardized beta weights to standardized weights)
Since 4 tests are planned, the alpha would be set at .0125 (.05/4) via the Bonferroni correction. Should the adjusted alpha be also applied to the p-values for the Wald tests in the "variables in question" output?
Thank you in advance!
FWER (Family-Wise Error Rate) corrections, such as the Bonferroni correction, can be applied to the p-values in the output of a logistic regression analysis. FWER corrections are used to control the probability of making at least one type I error (rejecting a null hypothesis that is actually true) when multiple tests are performed simultaneously.
When conducting a logistic regression, each predictor variable is tested for its association with the outcome variable. By default, each predictor variable is tested independently and the p-value for each predictor variable is reported.
When applying a FWER correction such as Bonferroni, the significance level for each test is divided by the number of tests performed. This helps to control the overall probability of making a type I error. For example, if the desired significance level is 0.05 and there are 10 predictor variables in the logistic regression, the corrected significance level for each test would be 0.005 (0.05/10). Any predictor variables with a p-value less than 0.005 would be considered statistically significant.
It's important to note that FWER correction become very conservative when the number of predictor variables is large, which can result in a higher rate of type II errors (failing to reject a null hypothesis that is actually false) . So, it is not always the best choice in all situations, and other methods like False Discovery Rate (FDR) correction should also be considered.
Its essentially analogous to ANOVA or ANCOVA in this context so logically one should consider this. However:
- with three categories you get complete protection against the partial null hypotheses if the overall test of the effect (probably the LRT for the factor analogous to the F test). This is the logic of Fisher's LSD and so no correction would be needed if you relied on the overall test of the effect for Type I error protection.
- with four or more categories this property no longer holds, but its not a good idea to use the Bonferroni correction as its rather conservative. I'd suggest a modified Bonferroni procedure such as the Hommel or Hochberg correction. Hochberg can be done by hand easily, but it and the Hommel correction is also implemented in R with the p.adjust() function. As the input is just a set of uncorrected p values you can use output from R or another package very easily:
> p.vals p.adjust(p.vals, method='hochberg')
This uses Wright's adjusted p value approach rather than altering alpha, but the decisions are equivalent:
Wright, S. P. (1992). Adjusted P-values for simultaneous inference. Biometrics, 48, 1005–1013. doi:10.2307/2532694.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Probability
- Probability Theory
- Reliability