I have four groups, the anova accepted the hypothesis of equal means. But when I used the two sample t-test 6 times (4 combination 2), two of these 6 tests are significant, which method we prefer?
The ANOVA (as any other test as well) never accepts a hypothesis. It will only tell you if you have enough data to reject a hypothesis, so that the interpretation of the sign of a coefficient or the decreased sum-of-squares as being expected for the population is (believed to be) warranted.
So the only result from the "non-significant" ANOVA is: you don't have enough data to conclude that the decrease of the sum-of-squares you observed in your sampmle should be expected in the population.
If you have several samples, and the hypothesis of equal variances makes sense, then it is highly recommended to estimate the variance from all the data you have ("pooled variance") and to use this for comparing individual samples. This pooled variance is estimated during the ANOVA, and it is used for post-hoc tests. It is not recommended to perform independent t-tests between groups, because here each test estimates the variance only from the two samples used in the respective t-test, so only a subset of the available information is used to estimate the variance., leading to less stable estimations with unneccesary large fluctuations of the variance stimates, the standard errors, the t-statistics and, eventually, the p-values.
In short: using the pooled variance is more robust, and robust methods should be preferred.
This is different is the assumption of equal variances is not reasonable. But then you would not have performed ANOVA anyway, and it is usually better to first understand why the variances are not equal and how this can be properly modelled instead of irgnoring it or using tests that are less sensitive to heteroscedasticity - but test entirely different hypotheses (you might not be aware of and also not interested in).
- Badges
- Science method