ANOVA is not interesting (and sensible) at all when you anyway want to compare individual groups. ANOVA is only relevant to asses the impact of a predictor (or a set of predictors) in a more complex model. It is only very, very indirectly and marginally related to the question if expected values differ between levels of a categorical predictor. Almost all post-hoc tests control the FWER independent of the ANOVA, and the "two-step-procedure" (first check ANOVA, then do post-hoc-tests) leads to additional problems.Note that "post-hoc" doe not mean "after ANOVA" but "after knowing the data (of the other groups)". This is often misinterpreted in many textbooks.

To my experience, differences between the different post-hoc tests are very "academic" and typically of only little practical relevance. Practically, there are only two tests I consider:

• Tukey's HSD for all-pairwise comparisons and
• Dunnett's procedure for multiple-to-one comparisons.

They are quite generally applicable and have no severe problems (as far as I know). Dunnett is superior for multiple-to-one comparisons, because Tukey assumes that k*(k-1) tests are performed (each of which can produce a false positive result) whereas Dunnett's procedure has to consider only k-1 tests (k is the number of groups).

There is an intermediate between Tukey and Dunnett: Hsu’s multiple comparisons with best (MCB). This test is like Dunnett's test but without a defined control group. This test first determines the "control group" as the "best group", which is the group with the most extreme mean and then compared all other groups to this one. It thus makes (k-1) tests (better than Tukey) but first has to determine the "best", what is negatively affecting the FWER (worse than Dunnett). (Note that selecting the "best" group by eye and then do Dunnett's test irgnores the uncertainty associated with the selection of the "best" group, so the FWER is not really controlled then).

If I test only very few and specific pairs of many possible pairs, I prefer Holm adjustments to control the FWER. This can outperform Tukey's test w.r.t. power.

For screenings it is often more sensible to control the FDR, what is done by the procedure of Benjamini/Hochberg.

However, in research it is more important - to my opinion - to think if and what error rate should be reasonably controlled at what level. The rather mindless control of a FWER at 5% is quite stupid, although typical...

ANOVA is not interesting (and sensible) at all when you anyway want to compare individual groups. ANOVA is only relevant to asses the impact of a predictor (or a set of predictors) in a more complex model. It is only very, very indirectly and marginally related to the question if expected values differ between levels of a categorical predictor. Almost all post-hoc tests control the FWER independent of the ANOVA, and the "two-step-procedure" (first check ANOVA, then do post-hoc-tests) leads to additional problems.Note that "post-hoc" doe not mean "after ANOVA" but "after knowing the data (of the other groups)". This is often misinterpreted in many textbooks.

To my experience, differences between the different post-hoc tests are very "academic" and typically of only little practical relevance. Practically, there are only two tests I consider:

• Tukey's HSD for all-pairwise comparisons and
• Dunnett's procedure for multiple-to-one comparisons.

They are quite generally applicable and have no severe problems (as far as I know). Dunnett is superior for multiple-to-one comparisons, because Tukey assumes that k*(k-1) tests are performed (each of which can produce a false positive result) whereas Dunnett's procedure has to consider only k-1 tests (k is the number of groups).

There is an intermediate between Tukey and Dunnett: Hsu’s multiple comparisons with best (MCB). This test is like Dunnett's test but without a defined control group. This test first determines the "control group" as the "best group", which is the group with the most extreme mean and then compared all other groups to this one. It thus makes (k-1) tests (better than Tukey) but first has to determine the "best", what is negatively affecting the FWER (worse than Dunnett). (Note that selecting the "best" group by eye and then do Dunnett's test irgnores the uncertainty associated with the selection of the "best" group, so the FWER is not really controlled then).

If I test only very few and specific pairs of many possible pairs, I prefer Holm adjustments to control the FWER. This can outperform Tukey's test w.r.t. power.

For screenings it is often more sensible to control the FDR, what is done by the procedure of Benjamini/Hochberg.

However, in research it is more important - to my opinion - to think if and what error rate should be reasonably controlled at what level. The rather mindless control of a FWER at 5% is quite stupid, although typical...

Jochen has given an excellent answer. I'll just add that my go-to is the Holm adjustment. The Benjamini/Hochberg can be somewhat more powerful when it's appropriate, but the Holm is delightfully easy to implement and not restricted to comparisons using identical testing strategies.

Patrick, just one or two quick comments, it might be clear, but anyway: Holm controls the FWER, whereas BH controls the FDR. This is a different concept and not just interchangeable. Which one to use depends on the aim (particularily on which kind of error rate should be controled). The difficuly in the implementation of a test should not matter. If I want to control the FDR but I don't know how, then I should find a way to learn/to do it instead of performing some other task I did not intend to do (this is funnily quite similar to the common habit that people switch from t-test to U-test when they observe difficulties with the assumptions). Apart from this BH is not really difficult to use even in EXCEL, and statistical software typically provide BH-adjustment or selection based on FDR anyway.

Ah, thanks, Jochen. And fair points. Can you point me to a reference on FWER vs. FDR, please, so I can learn from this? I'm not even sure what FDR stands for in this context.

Thanks!

Thanks for your answer.

I can not argue here, you seems to have a good experience of post-hoc analysis.

Anyway, after reading your answers I would add a supplementary question :-).

Are other post-hoc test useless ?

For example, I'm sure that I read somewhere that SNK were more powerful than HSD with an equality of observation in few groups...

Why a consent doesn't exist on the choose of one or two post-hoc test ?

(Except yours, on HSD and Dunnett's procedure) :-)

I agree that Jochen has given an excellent answer (as usual).  I would like to add though, that when one has 3 independent groups, Fisher's LSD can (and arguably should) be used, because with 3 groups, it does control the FW error rate at the per contrast alpha (see link below), and will be more powerful than the usual alternatives. 

Note too that it is the one common MC procedure that ~does~ entail a two-step procedure.  Step 1:  Check to see if the omnibus F-test is statistically significant.  If it is, proceed to Step 2:  Carry out all pairwise comparisons of the 3 groups using a modified t-test (with the MS_error from the ANOVA in the denominator of the t-test rather than a variance term based only on the 2 groups being compared). 

For more on why Fisher's LSD controls FW error with 3 groups, see the excellent chapter on MC procedures in Dave Howell's book "Statistical Methods for Psychology".

HTH.

http://www.ncbi.nlm.nih.gov/pubmed/17128424

@Cyril: As absolutely most related to statistical analysis, there is no doubt about the mathematical correctness and the preformance under clearly specified conditions - -  but not at all about the apropriateness of any method for any particular real-life problem. Every experiment has its own specific properties -most of them remain unknown!- and so there might be some better and some worse alternative for any purpose, and they may be different for each and every experiment. Nobody can know. And, as I already said: much of the differences are academic, and they are easily negligible in front of differences due to experimental specialities. So one is looking for some broardly applicable general-purpose solution, and there it turnes out that the mentioned tests find a common acceptance and other tests loose significance for practioniers.

A consent on the applicability of statistical methods won't be sensible, and it cannot be desired (if doing science is the aim). Too much depends on the particular features and properties and conditions of the experiment and the way the data was obtained. A very bad and harmful example for such a "consent" already exists: the common practise to call results "significant" when p

@Patrick: FDR = false discovery rate, the expected proportion of false-positives among the rejected null hypotheses.

The FWER is the expected proportion of "at least one false positive" in a family of tests.

Literature:

Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B 57, 289–300.

Benjamini, Y., and Yekutieli, D. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics 29, 1165–1188.

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics 6, 65–70.

Hommel, G. (1988). A stagewise rejective multiple test procedure based on a modified Bonferroni test. Biometrika 75, 383–386.

Hochberg, Y. (1988). A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–803.

Shaffer, J. P. (1995). Multiple hypothesis testing. Annual Review of Psychology 46, 561–576. (An excellent review of the area.)

Sarkar, S. (1998). Some probability inequalities for ordered MTP2 random variables: a proof of Simes conjecture. Annals of Statistics 26, 494–504.

Sarkar, S., and Chang, C. K. (1997). Simes' method for multiple hypothesis testing with positively dependent test statistics. Journal of the American Statistical Association 92, 1601–1608.

Wright, S. P. (1992). Adjusted P-values for simultaneous inference. Biometrics 48, 1005–1013. (Explains the adjusted P-value approach.)

Thank you Jochen for your passionate answer !

I'm aware that my question were a little provocant.

I must agree with you on the gold value : 0.05. That depend for sure of the situation, sometimes I wouldn't take that much error rate.

Concerning my "global statistic procedure" :

ANOVA -> Yes I do

Run a post-hoc -> Yes I do

Run a test at 5% or 1% because it is what want editor, reviewer and co-authors -> Yes I do

However

Do everything I can to improve the test power (data transformation, test choice...) -> Yes I do, because "all others" always talk about alpha but never about beta.

Probably (?) because the type II rate can be evaluated but not controlled as the type I rate does.

Finally you said that I ask "which post-hoc test "all others" use, to do the same again". Initially, I just wondered why there is that much of post-hoc procedure and when and why and 'with what' user compute them.

"Do the same again" ? No, I'm almost sure that SNK work fine with my data... but what will be my data tomorrow ? Who knows ?

Thank you all for your insights, for sure I'll have a close look to my data before running a new post-hoc procedure.

Thank you, Jochen. The source I'd read contrasting Holm and Benjamini-Hochsberg didn't draw that distinction, to my recollection. I appreciate the correction and will go read!

EDIT: Ah, and the Shaffer review is freely available!

http://wexler.free.fr/library/files/shaffer%20%281995%29%20multiple%20hypothesis%20testing.pdf

Jochen, I followed up on these references and feel on much firmer ground, thank you!

Jochen answered the question as asked. He also supplied the best advice:

Instead of looking for a (eventually non-sensical) consent about analysis methods we should/must start to think about the experiments and the data and its information ...

The clever and sensible choice of methods (including e.g. the choice of alphas and betas for tests) is the difficult part, because this depends strongly on the whole environment.

Post hoc is not the time to look around for tests. The tests and the necessary error rate must be selected ex ante.