ANOVA says there is a significant interaction (A*B) effect but when I run Tukey, factor A does not have significant effect on any level of factor B. However, sometimes ANOVA says there is a non-significant interaction effect but the Tukey shows significant effect of factor A on some levels of factor B. How can I explain this?
It can happen, and your supervisor is wrong.
The tested hypotheses are different, and there is no connection between the p-values obtained by a post-hoc test and the p-value from an Anova F-test. These are different things. The F-test tells you that the observed data is unexpected under a model lacking the interaction term. The post hoc tests evaluate the (un-)expectedness of the data under the model with one particular coefficient (of the inetraction) at a time restricted to zero, and Tukeys procedure further intends to control the family-wise type-I error rate.
The interesting question is what you call "significant". For instance, why should you consider the same "significance level" for the F-test and for the post-hoc tests? And is a control of a family-wise error-rate desired? And at what level (and why)?
A significant omnibus test is not a perfect predictor of what will happen you conduct follow-up tests. For example, we can have a significant omnibus F test in a one-way ANOVA but have no significant pairwise comparisons, or have a nonsignificant omnibus F test and have significant pairwise comparisons (even with multiplicity control).
The best way to follow up a significant interaction (especially a two-way interaction) is to conduct interaction contrasts. This evaluates all possible 2x2 interactions and will provide an answer regarding the nature of the interaction (based on which 2x2 interactions are significant).
Multiplicity control (e.g., Bonferroni, Holm) is controversial in post hoc analyses of interactions. If each of your contrasts can be viewed as an important null hypothesis test with inferential implications then you should impose multiplicity control. On the other hand, if you are only trying to summarize the nature of the interaction then often multiplicity control is not necessary.
Rob
I think you should be able to get at least one pair of significantly different means if ANOVA is significant no matter whichever mean separation procedure you are using. You may not be able to see clear segregation among your means.
It can happen, and your supervisor is wrong.
The tested hypotheses are different, and there is no connection between the p-values obtained by a post-hoc test and the p-value from an Anova F-test. These are different things. The F-test tells you that the observed data is unexpected under a model lacking the interaction term. The post hoc tests evaluate the (un-)expectedness of the data under the model with one particular coefficient (of the inetraction) at a time restricted to zero, and Tukeys procedure further intends to control the family-wise type-I error rate.
The interesting question is what you call "significant". For instance, why should you consider the same "significance level" for the F-test and for the post-hoc tests? And is a control of a family-wise error-rate desired? And at what level (and why)?
These cases occur repeatedly in ANOVA , and that did not refer to wrong result , because F- test and post-hoc test represent different test hypotheses, and you can perform post-hoc without ANOVA by using pooled variance.
Good Luck
Remember to test the assumptions of ANOVA to confirm the suitability of the statistical tool.
This is a common feature observed, While subjecting the data with ANOVA. When comparing between all the groups in a pooled manner, F value may not show statistical significance at 1% level or 5% level. But the post-hoc test may reveal statistical significance at alpha=0.05 while comparing any of the two groups.
Nothing wrong in such kind of results.
You can explain that there exist statistical significance difference between those two groups which was revealed in post-hoc analysis.
Though the F-test may not be significant at either 5% level or 1 % level, the p-value might be nearer or slightly above 0.05. (Eg.0.06 or 0.07, implying that the confidence level is 94% or 93% instead of our classical significance testing at either 95% level or 99% level)
Dear Serma
I agree with you about nothing wrong in these results when we find difference between F- test and post-hoc results, but we do not need explanation about why they did not gave same results, because the two tests examine different hypotheses.
Best regards
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis