Hello, I have run a logistic model and I have four categories of a predictor variable. One option I have for comparing groups is comparing each group to every other possible group (like a Tukey's test). However, given the nature of my research questions, I am only actually interested in four out of the eight comparisons, so I lose power and stack on degrees of freedom when I test all comparisons, resulting in some of the pairs being insignificant although they were significant in the original model.
Is it acceptable to run the model a few different times and change the reference group to just get the comparison I am interested in or is this too liberal and increases my probability of a type error? Another method that may fit is using a planned comparisons/contrast matrix to isolate the effects of the pairs I think are meaningful.
Any advice from a stats-savvy person would be helpful. I'm curious to know how you'd approach it!
It is generally not recommended to run a logistic regression model multiple times and compare the results, as this can lead to multiple testing problems and increase the probability of making a type I error (false positive). This is because each time you run the model, you are essentially conducting a hypothesis test, and the more tests you conduct, the greater the probability that you will observe a significant difference by chance alone.
Instead of running the model multiple times and comparing the results, you can use planned comparisons or contrast coding to compare the groups of interest in a single model. This approach allows you to specify the comparisons of interest in advance and test them simultaneously, rather than conducting multiple tests.
To do this, you can use a contrast matrix to specify the comparisons you are interested in. For example, if you have four categories of a predictor variable and you want to compare groups 1 and 2, groups 1 and 3, and groups 2 and 3, you could use the following contrast matrix:
Group 1 Group 2 Group 3
1 1 -1 0
2 0 1 -1
This contrast matrix would produce three contrast coefficients, one for each comparison, which can be tested in a single logistic regression model. This approach has the advantage of being more efficient and less prone to error than running multiple models and comparing the results.
It is also important to consider the sample size when conducting statistical tests. A larger sample size will generally increase the power of the test and make it more likely to detect a significant difference between groups, even if the difference is small. Therefore, if you have a smaller sample size and you are concerned about power, you may want to consider using a more powerful statistical test such as a bootstrapped t-test or a non-parametric test such as the Wilcoxon rank-sum test.
- Badges
- Science topic
- Similar topics
- Geoscience
- Cartography
- Mapping