I found a statistically significant difference between a dichotomous outcome variable and an independent variable with four categories on the initial chi-square test. I performed post hoc tests (multiple paired-wise comparisons with a Bonferroni correction for p-value, a total of 6 comparisons) to explore the groups that contributed to the difference but end up with non-statistically significant outputs in all comparisons using the adjusted alpha value as a reference for significance.
Is it statistically possible to come up with such findings? If so why it becomes significant in the initial chi-square test, and to which group we could attribute the difference?
Hello Adane,
Yes, it is possible. What causes the apparently paradoxical results to occur?
1. Each of the pairwise tests is run at a more stringent alpha level than the overall test. Hence, they are less powerful (if all else was equal);
2. If you're partitioning a contingency table into smaller sections for the follow-up tests, then the per-section N is smaller. As the power of a chi-square test does depend on N (in addition to the magnitude of the effect), this by itself (e.g., before the Bonferroni adjustment) makes the pairwise test less powerful.
3. It's possible that the "difference" that exists among the four classes of your IV is not pairwise in form. Thus, relying on just pairwise follow-ups fails to detect the difference(s).
Good luck with your work.
Hello David Morse,
Thank you for your concise explanations and best wish. it is very insightful and I highly appreciated your contribution.
To link paired comparisons and total test, you can use the following correction/adjustment:
To get the Bonferroni corrected/adjusted p-value, divide the original α-value by the number of analyses on the dependent variable. The researcher assigns a new alpha for the set of dependent variables (or analyses) that does not exceed some critical value: α critical= 1 – (1 – α altered)k, where k = the number of comparisons on the same dependent variable.
- Badges
- Science topic
- Similar topics
- Analytical Chemistry
- Extraction