In data analysis, the anova test is non-significant but sometimes while performing further post hoc test two group means are observed to be significantly different. At such condition how should i interpret the result?
A great question
Please see these links.
You will find the answer to the question.
Article Understanding one-way ANOVA using conceptual figures
https://statisticsbyjim.com/anova/post-hoc-tests-anova/
1. Also pay attention to issues such as F and T statistics and alpha error.
2. Think about whether this happens to all types of Post hoc? The answer is no.
This usually happens in cases where the probability value of the ANOVA test is so-called borderline (ie close to the alpha significance level).
ANOVA and post-hoc tests are not testing the same thing from the data. So, you really cannot use the result of one to make a decision about the result of the other.
This is actually quite a complicated question to answer. It depends very much on the number of means you have and the type of post hoc test.
Not all post hoc tests require a significant omnibus F test. Some are designed to protect familywise Type one error only if the omnibus F test is statistically significant. The best known example of this is the Fisher LSD test. However, it is a poor test in that it doesn't fully protect Type I error with with more than three means. Other tests (probably the majority of common post hoc procedures) don't require a significant omnibus F test. A good example here is the Turkey HSD test, but one could include many others - notably the Holm, Hochberg and Hommel corrections. The Hochberg and Hommel corrections are particular good choices for pairwise post hoc tests (as they are among the most powerful options whilst also protecting Familywise error properly).
As to why it occurs, that's also quite subtle. As Blaine Tomkins mentioned they are not testing the same hypothesis. The omnibus test is testing the overall pattern for any deviations among the means. By restricting the hypotheses to pairwise differences you reduce the scope of the test. The more general scope of the omnibus test has less statistical power than the focused pairwise tests. This is a general principle - vague hypothesis tests have less statistical power than precise ones.
There is a section in my book Book Serious stats: A guide to advanced statistics for the behavi...
on this, but it should be covered in most advanced statistics texts. Many introductory texts however get this wrong and incorrectly state that out posts hoc tests must follow a significant omnibus F test.
What is the consequence of following this incorrect advice? Very simply there is a loss of statistical power if you use a test like Tukey's HSD only following a significant omnibus F test. If you use a test like Fisher's LSD you may be OK but it could also inflate Type I error rates (e.g., with more than 3 means).
- Badges
- Science topic