Sure it can.

There is a misconception that the ANOVA is required for post-hoc tests and their error protection or that they are to be performed after an ANOVA. This is all essentially wrong.

The term "post hoc" comes from "after seeing the data", what partically means that information from all the data is used in each test, where actually only a subgroup of the data is used/compared. Specifically, it is the pooled variance estimate that is used to get better estimates for the standard errors of the pairwise differences (therefor the assumption of variance homogeneity has to be made!).

Another, actually only marginally related topic is the control of the error-rate, when the tests are used to "screen" for differences between the groups. Collecting "significant" differences means also collecting false-positives. The researcher has to decide how to handle this problem (if it is a problem at all). The most common remedies are the control of the error-rate with respect to the whole familiy of performed tests ("family-wise error-rate", FWER) what equals the "expected rate of selecting at least one false-positive", and the less conservative approach of controlling the rate of false-positives among the selected (the rejected null hypotheses), what is called the "false-discovery rate", FDR). Most post-hoc tests control the FWER.

When a post-hoc test does control the FWER, it does it on its own and not by the help of a "protecting" significant ANOVA. There is only one exception known to me: Fisher's LSD test, that compares 3 groups and in fact requires a significant ANOVA. Tukey's HSD test controls the FWER and it does not require an ANOVA.

So there might be two reasons for this misconception: 1) that Fisher developed a method that actually required the "ANOVA protection" and 2) that computer software typically first calculate an ANOVA before any post-hoc tests because some of the intermediat results (like the pooled variance) are readily calculated in the ANOVA algorithm and can then simply be re-used by the post-hoc tests.

There surely is a possible third reason: many stats courses and books refer to the ANOVA as a test for the "omnibus" hypothesis µ1=µ2=µ3=...=µ0

But in fact the ANOVA tests VAR(between) = VAR(within), what is really something different. I see that the often cited hypothesis µ1=µ2=µ3=...=µ0 is appealing and better understandable, and it may well serve as kind of a mental approximation or an argumentative simplification. That it is just a picture becomes obvious when the results of ANOVA and post-hoc tests do not match. This seems confusing for those who always thought that both hypotheses were the same.

To give a proof of example that my answer above ("yes") is correct I give a numerical example with analysis results (produced in R; alpha=0.05).

Sure it can.

There is a misconception that the ANOVA is required for post-hoc tests and their error protection or that they are to be performed after an ANOVA. This is all essentially wrong.

The term "post hoc" comes from "after seeing the data", what partically means that information from all the data is used in each test, where actually only a subgroup of the data is used/compared. Specifically, it is the pooled variance estimate that is used to get better estimates for the standard errors of the pairwise differences (therefor the assumption of variance homogeneity has to be made!).

Another, actually only marginally related topic is the control of the error-rate, when the tests are used to "screen" for differences between the groups. Collecting "significant" differences means also collecting false-positives. The researcher has to decide how to handle this problem (if it is a problem at all). The most common remedies are the control of the error-rate with respect to the whole familiy of performed tests ("family-wise error-rate", FWER) what equals the "expected rate of selecting at least one false-positive", and the less conservative approach of controlling the rate of false-positives among the selected (the rejected null hypotheses), what is called the "false-discovery rate", FDR). Most post-hoc tests control the FWER.

When a post-hoc test does control the FWER, it does it on its own and not by the help of a "protecting" significant ANOVA. There is only one exception known to me: Fisher's LSD test, that compares 3 groups and in fact requires a significant ANOVA. Tukey's HSD test controls the FWER and it does not require an ANOVA.

So there might be two reasons for this misconception: 1) that Fisher developed a method that actually required the "ANOVA protection" and 2) that computer software typically first calculate an ANOVA before any post-hoc tests because some of the intermediat results (like the pooled variance) are readily calculated in the ANOVA algorithm and can then simply be re-used by the post-hoc tests.

There surely is a possible third reason: many stats courses and books refer to the ANOVA as a test for the "omnibus" hypothesis µ1=µ2=µ3=...=µ0

But in fact the ANOVA tests VAR(between) = VAR(within), what is really something different. I see that the often cited hypothesis µ1=µ2=µ3=...=µ0 is appealing and better understandable, and it may well serve as kind of a mental approximation or an argumentative simplification. That it is just a picture becomes obvious when the results of ANOVA and post-hoc tests do not match. This seems confusing for those who always thought that both hypotheses were the same.

To give a proof of example that my answer above ("yes") is correct I give a numerical example with analysis results (produced in R; alpha=0.05).

There are numerous situations where an ANOVA procedure identifies a significant difference but none are identified in post-hoc analysis.

One situation that may arise is when the post-hoc analysis is too conservative. This might come from using something like the Bonferroni test or result from the sample size (e.g. if it's too small and thus yields limited power). The LSD test, however, is exceedingly liberal so this issue may not apply for the case you've brought up.

The other situation, which is quite common in my experience, arises when running a more complex ANOVA - repeated measures, especially. Remember that the ANOVA comparisons, whether between subjects or within, provide comparisons reliant on different means combinations. Therefore, significant differences may appear between the different means combinations but not come-up when performing direct pair-wise comparisons between the individual groups.

One way I've gotten around this sort of issue when dealing with a complex data set with multiple factors is to run an initial complex ANOVA procedure (e.g. multivariate or repeated measures as appropriate), identify which factors have significant differences, and then run separate, simpler ANOVA on those factors alone. Yes, this requires using a Bonferroni correction to ensure appropriate alpha. However, it often helps better reveal exactly what's happening in the data to account for the differences.

I hope this helps!

A few precisions and comments, on Jochen's post

1) The Fisher LSD, strictly speaking, does not require a significant ANOVA to be made ; it obviously does not control the FWER. When performed only after a significant ANOVA, it is called protected LSD (PLSD), and still does not completely control the FWER (weak control, but no strong control). I guess quite often both are confused, and the protected one done when speaking of LSD.

2) Most of multiple-comparison tests (not necessarily post-hocs, as you well explained) are made after ANOVA because they share the same assumptions (normality, homoscedasticity and independance) and use the residual sum of squares and other ANOVA model results. But ANOVA is indeed just a tool to obtain these values, and its significance results are useless

3) Multiple-comparisons tests and ANOVA does not exactly test the same H0 region in the parameters space, so it can occur than one is significant and is the other not. Imagine one tests a circle centered on 0 (ANOVA) and the other a square of same area [probability] (multiple-comparisons) : in « corners », the two surfaces does not overlap, so depending on the position of the experimenal point, all can be significant [outside square and circle], not-significant [inside square and circle] or one is, the other not... Of course, there is never contradiction between this, because  not-rejecting H0 is not saying it is true, simply in some cases one method is more powerful than the other, in some cases the other way round...

4) Beware that there is in fact two one-way ANOVAs:

 - fixed effects, where H0 is indeed µ1 = µ2 = ... vs H1 at least one is different;

 - random effects, where H0 is much as you said (additional inter-variance is null, vs H1 additional variance > 0).

In both cases, the test criterion is exactly the same, hence the confusion, but when dealing with mode complex situations (two-ways), test criteria may change.

Note also that ANOVA is not simply a variance comparison, even is often presented like this, because you always make a unilateral test, compared to 0, and not a bilateral one, compared to 1...

Last, in the fixed effects framework, if assuming Gaussian distribution and same variance (ANOVA assumptions), then necessarily means are either equal (H0) or different (H1)

I don't think so that test of significance for anova is a misconception, If you decide work in an specific model, you must respect its restrictions, and the, sine qua non postulate is the significance of the parameters of the expected mean squares...

Edgar, I essentially agree. But you have to keep in mind that tests do nothing else but limiting the reate of false decisions. This task is accomplished by the posthoc-tests. The (ANOVA-)F-test also does it (there we agree), but from a different angle. Therefor it is possible that the conclusions/descisions from both "perspectives" (post hoc vs. ANOVA) may disagree. There is no need that these procedures are consistent. Now, since the identification the particular groups that are different is the aim, and ANOVA does not address this question, I wonder what ANOVA can add.

Anova and post-hoc, have different functions, you must use anova to validate your randomization process, and then, with the error estimated, you can use it for multiple comparisions. Your decision to use one or another post-hoc are independent of your randomization process. Although if you can demonstrate, that tukey can evaluate the effects of a model in an experimental design, it will be interesting...

> Edgar : I don't see how ANOVA can validate the randomization process, since having random values (for error) is a pre-requisite for using ANOVA (or any other statistical method)... And if can, how it would do it better than suited multiple-compararisons. However, that ANOVA gves the estimated error used in post-hoc or multiple-comparisons tests is true.

For Tukey and effect size: just try the TukeyHSD function in R, il gives confidence intervals on each difference (as does multcomp()). But may be you have something different in mind when speaking of "effect of a model" ?

The discussion here gives me new knowledge. I have not experienced a situation where a significant ANOVA value does not give a significant post hoc test or vice versa. All the same, thanks to all the contributors. 

Emmanuel, you can find a good answer to your question in Cochran y Cox, 1957 (block efficiency), or even, Tukey, in 1949, develop a test for validate a model in the anova context (additivity). I insist, anova and pos-hoc test have different functions in the experimental design. Good Night

Well, I have come across this problem, and do not know how to deal with it. 

Could you please give me some advice?

Hi Zhe,

Could you provide more information about the type of ANOVA you're running? How many factors are you comparing? Is it one-way or repeated measures? How many levels are in each factor? Are you looking at main effects or interaction effects?

There are many approaches. The type of approach used will depend on what you're comparing.

Dear colleagues

In descriptive statistics any set of data must describe with parameter from central tendency ( mean ) and parameter from dispersive tendency ( std or stderr ) because one of them not enough to describe the data for example the two set of data ( 1 , 5 , 54 ) and ( 18 , 20 , 22 ) have the same mean 20 but they have different variation and range among values .

In inference statistics , the test of significant differences among set of means by using Duncan multiple test or Revised LSD included both the value of mean , variation between means and variation within ( mse ) . So I confirm the comments of other colleagues , that we can perform post hoc test with out ANOVA and we can use pooled variance instead of mse .

The ANOVA and multiple comparisons test presented different test and in some cases gave us different decisions like non significant in F test ANOVA ( for the variance ) and significance differences with Duncan test for the means .

Good Luck

Dear friend 

If  one way or two way anova used according to your data. If one control and other treated group used dunnet test applied. It is necessary to use appropriate statistics before starting the experiment?