If the means of several (2+) groups/samples are compared, many people perform an ANOVA to test (and hopefully reject) the H0 that all samples come from populations with the same means. Then it must be identified which samples may be from populations with unequal means, so pairwise comparisons are conducted, often named "post-hoc tests". This procedure is followed very often and described in numerous textbooks. But I can't find a reference to where such a procedure (post-hoc testing) was first proposed for multiple testing problems. Was it Fisher ("Fisher's LSD"; I could not find anything in his famous books "Research methods" and "Design of Experiments", but maybe I have overseen something)? And further, who said that the omnibus-test (rejecting the global H0 in the F-test) is a pre-requisite to adjust the family-wise error-rates? Would be great to have some references. Right now, for me, it looks like all this was fallen down from the skies or just "god-given"...
The motivation is to control the family-wise error rate at a fixed level such as 5% (see the literature on "gatekeeper" tests).
That does not cohere with Fisher's interpretation of the p-value as the strength of evidence against the null hypothesis.
Thanks, David. But that is not answering my question. Who came up with this idea? Who first promoted to
* first do an ANOVA, and if it is "significant"
* do post-hoc tests to find pair-wise differences
?
Jochen,
I do not have the answer (yet), but I believe that Scheffé and Tukey are great candidates, because their papers are from the begining of 1950's. Actually, I already looked for the original Tukey's paper, but only found a reference to a "unpublished note"...
Let's keep looking for...
Miller's book -- http://tinyurl.com/jwgbo8a -- on p. 90 attributes the basic idea to Fisher's Design of Experiments but without any page or chapter citation.
Design of Experiments (1935) was written before Fisher's thought developed in response to Neyman. The FWER control idea fits Neyman's theory.
You're welcome. I have the first edition, not the edition of the URL. It is in the section called "Least significant difference test (Fisher)."
ANOVA is an omnibus test, which means that it tests for an overall experimental
effect: so, there are things that ANOVA cannot tell us. Although ANOVA tells us whether the experimental manipulation was generally successful, it does not provide specific information about which groups were affected.
Post hoc tests consist of pairwise comparisons that are designed to compare all different combinations of the treatment groups. So, it is rather like taking every pair of groups and then performing a t-test on each pair of groups.
However, pairwise comparisons control the familywise error by correcting the level of significance for each test such that the overall Type I error rate (α) across all comparisons remains at .05. There are several ways in which the familywise error rate can be controlled. The most popular (and easiest) way is to divide α by the number of comparisons, thus ensuring that the cumulative Type I error is below .05.
.. In other words the one-way independent ANOVA compares several means, when those means have come from different groups of people. For example; if you have several experimental conditions and have used different participants in each condition. When you have generated specific hypotheses before the experiment use planned comparisons, but if you don’t have specific hypotheses use post hoc tests.
There are lots of different post hoc tests: when you have equal sample sizes and homogeneity of variance is met use REGWQ or Tukey’s HSD. If sample sizes are slightly different then use Gabriel’s procedure, but if sample sizes are very different use Hochberg’s GT2. If there is any doubt about homogeneity of variance use the Games–Howell procedure.
Therefore, when considering which post hoc procedure to use we need to consider three things: (1) does the test control the Type I error rate; (2) does the test control the Type II error rate (i.e. does the test have good statistical power); and (3) is the test reliable when the test assumptions of ANOVA have been violated?
I hope this addresses the question in some way. Cheers
..as for references
Howell, D. C. (2006). Statistical methods for psychology (6th ed.). Belmont, CA: Duxbury. (Or you might prefer his Fundamental Statistics for the Behavioral Sciences, also in its 6th edition, 2007.
Both are excellent texts that provide very detailed coverage of the standard variance approach to ANOVA and the GLM approach)
Iversen, G. R., & Norpoth, H. (1987). ANOVA (2nd ed.). Sage university paper series on quantitative applications in the social sciences, 07-001. Newbury Park, CA: Sage. (Quite high-level, but a good read for those with a mathematical brain.)
Klockars, A. J., & Sax, G. (1986). Multiple comparisons. Sage university paper series on quantitative applications in the social sciences, 07-061. Newbury Park, CA: Sage. (High-level but thorough coverage of multiple comparisons)
Azubuike, thank you for these long posting... but are completely off-topic. I asked for the ORIGIN of the idea that post-hoc tests are related to Anova and should be done together with Anova. I did NOT ask for things Anova can tell us and things Anova won't tell us, I neither wanted to know how many different kinds of post-hoc tests for wich occasions are available. My question was staright and simple: who was first proposing this strange association between the omnibus test and the -anyway usually nessecary- pairwise comparisons. Fisher, who invented the Anova, was clearly not the on who would have proposed the need for the control of a FWER, and the intention of Anova is -intelecually and purposefully - something completely different than testing µ1=µ2=µ2...=µn. It is rather a procedure to get a hint if keeping the whole predictor in the model is useful.
Okay Wilhem. I supposed 'the origin of ANOVA' in your question meant the line of thought that justifies the relationship between ANOVA and post hoc.
Cheers
Parolini, G. (2015). The Emergence of Modern Statistics in Agricultural Science: Analysis of Variance, Experimental Design and the Reshaping of Research at Rothamsted Experimental Station, 1919–1933. Journal of the History of Biology, 48, 301-335.
Thank you Craig, but I could not find any hint in there how post-hoc tests are related to ANOVA. However, it explains nicely how ANOVA is thought to investigate different sources of variation together, and, as I understood, to eliminate the need of pair-wise comparisons, where they are not of importance.
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Modeling
- Analyses of Variance
- ANOVA