I want to understand the basic differences among various post-hoc tests such as Bonferroni-correction, Tukey, LSD, Scheffe etc. in terms of their power and application.
https://www.uvm.edu/~dhowell/methods9/Supplements/OldChapter12.pdf
Article Statistical notes for clinical researchers: Post-hoc multipl...
Article Multiple comparison analysis testing in ANOVA
The sources linked to by Jochen Wilhelm are really good.
Let me a offer a few pieces of advice, or things to look out for.
1) Be sure to pay attention to the assumptions of the methods. For example, if they require equal sample sizes or groups with equal variances. This is covered somewhat in these documents, but good to keep in mind.
2) Be aware that there are variants on some of these methods, and authors are not always clear which they are talking about. For example, Tukey-Kramer is a variant of Tukey HSD, but a text may not be clear which one they are talking about. The RDE document has pretty good list and gives some sense of how these methods are related. This is valuable.
2b) Be sure to understand which method software is using. It is sometimes difficult to find documentation of the specific implementations a software package is using, and sometimes the documentation isn't clear. Again, look at the list in RDE. If something is called "Holm" or "Sidak" or "Bonferroni", are we sure which method they are using?
I would try very simple explanation:
Bonferroni correction is just an adjustment of p-values for paiwise testing of differences between levels.
Example: You have three levels or groups (A, B, C). You can test for differences between means of two of them for example by t-test, that means you need three t-tests (A-B, A-C, B-C). The thing is, that you need to adjust "alpha", because you have three tests, which need to have desired error rate together, so you can reject null hypothesis, when p < alpha/number of groups. There are more methods on this, which were originally used for testing for differences between vector means after Hotelling T-test or MANOVA.
Using Scheffe or Tukey HSD test (HSD stands for "Honest significant difference") is different approach. Tukey HSD test is for pairwise comparisons group means based on studentized range distribution, Scheffe's method is using F distribution for such pairwise comparisons.
Conclusion: If you need pairwise copmparisons of group means after ANOVA, use TukeyHSD test, as it was shown to be most powerful. There are some (I think very rare) occasions, when Sheffe's method could be better - you can create such datasets, if you are interested, I was trying that one time and it is possible, but definitely not common :-)
If you need multiple comparisons after Kruskal-Wallis test (non-parametric ANOVA), you need to use specific pairwise comparisons for that. In R, it could be done by function "kruskalmc" from package "pgirmess".
Good luck!
You can always apply the Games-Howell post-hoc test, as this test does neither assume a normal distribution of the values (homogeneity of variances), nor equal sample sizes.
is there any summary table of all post hoc test? (when and where can we use them)
- Badges
- Science topic
- Similar topics
- Methodology
- Research Methodology
- Research Papers