Answer:

Tukeys is the more reliable answer, given the population variances are similar. A possibly sligtly better appraoch for your setup was Dunnett's test (developed especially for multiple-to-one comparisons).

Explanation:

When you do Tukeys test, the variance is estimated from the whole set of data (from all 4 groups) as a pooled estimate. If the population variances are the same in all groups, such a pooled estimate is much more robust and precise than the individual estimated from just a part of the whole set of data. Further, Tukeys procedure adjusts the p-values for multiple testing, so that the family-wise error rate is controlled (probability to get at least one false positive among the family of tests performed).

The individual t-tests, in contrast, can estimate the variances only from 2 of the four groups, what is less precise and less reliable. Further, the p-values are not corrected for multiple testing, so there is no control of the family-wise error rate. Considering results as "significant" at the usual level (5% typically) will give you almost a 15% chance of having at least one false positive when the all 3 null hypothesis were true (=1-(1-alpha)^k, where alpha=significance [0.05] and k=number of tests [here:3]).

I think the main difference here is that with your t-test you're doing a single comparison. The Tukey method however deals with all possible pairwise comparisons, while a the same time correcting for the increase in type 1 error that you would otherwise have when doing multiple contrasts.

So the t-test in your output is just a single comparison.

That same contrast with Tukey would be a contrast given that

you are doing a whole bunch of other contrasts and thus correcting

for type 1 error, because each contrast has a chance of type 1 error

and these would just pile up with every contrast that is made.

http://en.wikipedia.org/wiki/Tukey%27s_range_tes

Answer:

Tukeys is the more reliable answer, given the population variances are similar. A possibly sligtly better appraoch for your setup was Dunnett's test (developed especially for multiple-to-one comparisons).

Explanation:

When you do Tukeys test, the variance is estimated from the whole set of data (from all 4 groups) as a pooled estimate. If the population variances are the same in all groups, such a pooled estimate is much more robust and precise than the individual estimated from just a part of the whole set of data. Further, Tukeys procedure adjusts the p-values for multiple testing, so that the family-wise error rate is controlled (probability to get at least one false positive among the family of tests performed).

The individual t-tests, in contrast, can estimate the variances only from 2 of the four groups, what is less precise and less reliable. Further, the p-values are not corrected for multiple testing, so there is no control of the family-wise error rate. Considering results as "significant" at the usual level (5% typically) will give you almost a 15% chance of having at least one false positive when the all 3 null hypothesis were true (=1-(1-alpha)^k, where alpha=significance [0.05] and k=number of tests [here:3]).

Consider why we use Tukey's test at all. Tukey's HSD was designed to deal with familywise error: simplistically, the more pairwise comparisons you make the more chances you have to make a type I error. Tukey's test was designed to ensure that (given certain assumptions hold) the FWE will be at most whatever the chosen alpha level is, making it a kind of t-test that takes FWE into account. ANOVA is (ostensibly) another way to avoid FWE by avoiding multiple tests through a general ratio using all group means together. So why do we use both? As you state, ANOVA doesn't tell us where the (supposed) significance lies, because we avoided multiple tests by combining the means. However, as the entire point to Tukey's test is to correct for FWE when making multiple tests, what's the point of the ANOVA F statistic? In general, a decrease in power increased chances for error (see e.g., https://marces.org/EDMS/fac/Hancock/writings/HancockKlockars1996.pdf and Introduction to Robust Hypothesis Testing by Wilcox). Also:

"Many of the statistical methods routinely used in contemporary research are based on a compromise with the ideal (Bakeman et al., 1996). The ideal is represented by permutation tests, such as Fisher’s exact test or the binomial test, which yield exact, as opposed to approximate, probability values (P-values). The compromise is represented by most statistical tests in common use, such as the t and F tests, where P-values depend on unsatisfied assumptions. In this book, an assortment of permutation tests is described for a wide variety of research applications. In addition, metric distance functions such as Euclidean distance are recommended to avoid distorted inferences resulting from nonmetric distance functions such as the squared Euclidean distance associated with the t and F tests." (Mielke, P. W., & Berry, K. J. (2007). Permutation methods: a distance function approach (2nd Ed.). Springer.) Permutation versions of ANOVA exist and are superior.

Tukey's test was designed to correct for multiple tests by itself, using it after using another tests negates (to some extent) the entire reason for Tukey's test in the first place. Using multiple t-tests just undoes what ANOVA avoids and Tukey's test accounts for: reduction of FWE, and both ANOVA and t tests tend to have the same problems (like outliers). An assumption of ANOVA is homoscedasticity. Heteroscedasticity, or unequal variances among groups, increases the chances of type one error. The F-test is, after all, a ratio of variance among groups, so it doesn't control type I error well when distributions differ among the groups. That could explain why ANOVA yielded a significant result but Tukey's test did not (although homoscedasticity is also an assumption of the latter test). Statistics based on trimmed means or medians rather than all three (ANOVA, t tests, and Tukey) are better as they control for outliers and are robust to violations of homoscedasticity. It's hard to know what to recommend without knowing more about the data and methods, but I can say

1) As already said by others, using Tukey's test rather than t-tests for more than two groups is definitely advisable

2) You don't need to use ANOVA and Tukey's test. You can just use Tukey's test and it's actually worse in general to use both.

Hi Ruth, Multiple t-tests lead to an increased alfa-error or type I (take a look to Bonferroni correction). Post-hoc ANOVA tests should only be performed when the ANOVA “p” is less than your predefined alfa error (in your case 0.05). There are several flavours for ANOVA post-hoc tests (some more powerful than others, i.e. able to detect differences when they really exist, others less and unable to detect differences when in reality there are, and the other way round "concluding" that are differences when in fact they not exist...). As previous contributors stressed the homogeneity of the variability among groups may lead to choose a particular test and to different conclusions. Most probably “you are” in the grey area where the decision of “significant” or “non-significant” effect is skin-deep or a “flavour based choice” (I have experienced that the variability when dealing with animals is HUGE when comparing to pure analytical data). Maybe, if possible (I know it is not an easy solution – due to cost and to ethical constraints), a different experimental design (different doses, number of subjects) could lead you to a more robust/reliable decision. If possible, you can use the current data/knowledge to plan another set of experiments with a pre-defined power (based on the experimentally observed variability, alfa and number of subjects) and minimize the chance of getting into a “grey area” decision. Don’t forget to report the actual effect and the variability observed in the different groups.

Luis, you say "Post-hoc ANOVA tests should only be performed when the ANOVA “p” is less than your predefined alfa error (in your case 0.05)." - this is simply NOT TRUE. It is known since decades now that such "Anova-protected post-hoc tests" don't do a good job. Good posthoc tests control the error rate by themselves, a "protection" would only decrease the power. There are two exceptions for posthoc tests that in fact need a protection: one is Fishers LSD, where the uncorrected p-values from the t-tests are taken, but this is valid for ONLY 3 groups and does not validly extend to more groups. I don't recall the second exception (SNK?) but that's not important, since there are better alternatives (eg. Tukey).

Dear Jochen, As FAIK post-hoc tests should only be performed when ANOVA’s p value are less than the pre-defined alfa-error. ANOVA’s p-value only tells you if there is a statistically significant “difference” between “any of the groups” – it compares a pooled residual MS with the MS of the factor under study – as a F-test ), obviously assuming that all ANOVA pre-requisites are assumed. This should be truth when you assume that the uncertainty associated to the measurements made ( i.e. are similar and you get a pooled (“average”) uncertainty) are similar). Please enlighten me if I’m wrong. BR Luis

Dear Luis:

With respect to your question, it is worth pointing out that a mass of literature exists questioning the legitimacy of null hypothesis significance testing in its entirety. For example:

"In a recent article, Armstrong (2007) points out that, contrary to popular belief, “there is no empirical evidence supporting the use of statistical significance tests. Despite repeated calls for evidence, no one has shown that the applications of tests of statistical significance improve decision making or advance scientific knowledge” (p. 335). He is by no means alone in arguing this. Many prominent researchers have now for decades protested NHST, arguing that it often results in the publication of peer-reviewed and journal endorsed pseudo-science. Indeed, this history of criticism now extends back more than 90 years (e.g., Armstrong, 2007; Bakan, 1966, 1974; Berkson, 1938; Boring, 1919; Campbell, 1982; Carver, 1978, 1993; Cohen, 1990, 1994; Edwards, 1965; Falk, 1998; Fidler, Thomason, Cumming, Finch, & Leeman, 2004; Fisher, 1955, 1956; Gigerenzer, 1987, 1993, 2004; Gigerenzer et al., 1989; Gill, 1999; Granaas, 2002; Greenwald, 1975; Hubbard & Armstrong, 2006; Hubbard & Lindsay, 2008; Hubbard & Ryan, 2000; Jones & Tukey, 2000; Kirk, 1996, 2003; Lindsay, 1995; Lykken, 1968, 1991; Meehl, 1967, 1978, 1990; Nunnally, 1960; Rosnow & Rosenthal, 1989; Rozeboom, 1960; Schmidt, 1992, 1996; Schmidt & Hunter, 1997; Sedlmeier & Gigerenzer, 1989; Skinner, 1972; Thompson, 1996, 1999, 2002, 2006, 2007; Tukey, 1991)… In his outgoing comments as editor of the Journal of Applied Psychology, Campbell (1982) wrote:

'It is almost impossible to drag authors away from their p values, and the more zeros after the decimal point, the harder people cling to them. It is almost as if all the statistics courses in the world stopped after introducing Type I error. …Perhaps p values are like mosquitos. They have an evolutionary niche somewhere and no amount of scratching, swatting, or spraying will dislodge them. Whereas it may be necessary to discount a sampling error explanation for results of a study, investigators must learn to argue for the significance of their results without reference to inferential statistics. (p. 698)'"

(http://psychology.okstate.edu/faculty/jgrice/psyc5314/SignificanceSorceryLambdin2012.pdf). For a comprehensive reference list see here: http://warnercnr.colostate.edu/~anderson/thompson1.html

Second, the reason not to perform a post hoc test do (as Joachim stated) reduce power and increase the chance for error is the very reason that we have ANOVAs rather than just performing multiple t-tests. For each test there exists a certain amount of likelihood for a Type I error, and thus the more tests one performs the more tests that do not attempt to correct for this represent an additional chance for such an error. ANOVA is very similar, as you no doubt are aware, to t-tests. The difference is entirely due to avoiding the kind of error (familywise error/FWE) that multiple t-tests would entail: the F statistic yields a value based on variation among all groups rather than pairwise comparisons between groups. As the only way a post-hoc test can be of value is by differing in the way it determines significance one is either basically double-dipping (if the F test is not "statistically significant") and regardless increasing the possibility of error. On this see e.g., Bernhardson, C. S. (1975). Type I error rates when multiple comparison procedures follow a significant F test of ANOVA. Biometrics, 31, 229-232.

Dear Luis, many people say this ("post-hoc tests should only be performed when ANOVA’s p is significant"). But I never saw a publication demnonstrating this(*). My search only revealed publications that demonstrated that this common belief is suboptimal, somtimes harmful, and never justified by logic. If you can reference a paper where this "common belief" is logically justified or deduced, I will change my mind. Otherwise I can only repeat my statement: ANOVA tests the reduction of the residual variance by adding one (or more!) predictor(s) to a model. When the predictor is binominal or quantitative, the result is logically, mathematically and practically identical to the t-test or the simple regression. This argument is often extended to multinominal predictors and it is said that the test of the variance reduction would be identical to the test of an "omnibus null" that all the expected values for all levels of the predictor are the same ("all means are equal"). This is - as far as I know - never theoretically demonstrated. As long as I find only papers demontrating the problems with this belief and no paper that soundly justifies it, I will call it an urban legend. Please enlighten me if I am wrong!

(*) Again, theer is ONE exception that might also be in part the reason for this confusion. It is coming from RA Fisher himself, who developed the ANOVA. He demonstrated that the FWER is controled for the family of standard t-tests (using the pooled variance estimate) if they are "protected", i.e. performed only after the ANOVA could reject the null that the residual variance is not altered by adding the predictor (Fisher's LSD). I never understoof why Fisher was concerned with this, since he strongly disagreed with the adherence to the control of error-rates as suggested by Neyman. But he probably needed this for his fiducial inference... I don't know.

Thanks Andrew and Jochen for your contributions (including the references). Regarding published papers I acknowledge the lack of references supporting my view (actually it is based on what I’ve been told in classes - long time go...) but I will try to dig down the subject. I will try to perform a simulation and will share the outcome asap. Nevertheless I think we might be, as I stressed before, in the grey area were you decide if the effect is significant or not based on the test you choose to support your decision. BR Luis

Dear Luis:

I applaud your approach and will be very interested to hear your results! Perhaps I am overly jaded, but I view any hypothesis testing as a "grey area" (like most of research) because whether one refers to testing hypothesis in the broad sense of what is essentially science or "hypothesis testing" (i.e., accept/reject null hypothesis) much depends upon the design of the experiment, the type of data collected and how it was analyzed, and so forth (all things I have no doubt you know). Nevertheless, were it not for statistical theory and the logic of experimental design, it wouldn't even be a grey area. The reason for supposing problems with post-hoc ANOVA currently has both a an experimental foundation (going back to the study I cited in '75) as well as a mathematical/logical one, and has even made its way into textbooks (two of which I believe I mentioned in an earlier response). The underlying reason for doubt seems fairly straight forward (and I linked to the most straightforward description I could find quickly that was freely available in the response I made in which I referred to the textbooks). Nonetheless, there is a paucity of empirical data as, and few studies that have tested the robustness of ANOVA with or without assessing whether its power is increased by a particular post-hoc analysis have done so without problematic assumptions (for example, empirical studies of post-hoc tests tend to compare these without testing post-hoc tests after ANOVA vs. without any post-hoc tests; the study I referred to is an important and major exception). Thus, we currently have little empirical data other than that which suggests post-hoc tests of a statistical method designed to avoid the errors inherent in pairwise t-tests of groups greater than 2 is not improved by an additional test that was designed to account for the bias of multiple pairwise testing in a way ANOVA doesn't (thus ensuring the two tests are not redundant and logically cannot but to some extent "undo" or otherwise bias the other, for if it did not then the tests would be identical and there would be no use for both). Logic, ironclad though it may be as an ideal, can be too precise for the world we live in, hence the importance of the simulations like those you intend to perform, A rather fundamental question, however, is why what using both an F statistic and Tukey's HSD can tell use that the latter alone cannot? For if it is not a reliable measure of within group variance the way that the t-test is, then it is no more use than using a t-test after one's ANOVA. And if it is reliable, then why use ANOVA? In particular, as both measure were known to be inferior by those that designed them but the "ideal" tests were not possible for computational reasons, why are we not using tests from the '30s that were then recognized by the same people that developed (or aided in the development of) modern statistics such as Fisher now that we can run the kind of tests they described but could not implement? In any event, I have long since drifted away from anything other than meaningless rambling (and have not slept in some time), so I will only repeat that any simulations you run would be of great use and most welcome.

There are several post hoc options in most cases. See the following for some discussion of the topic.

http://stats.stackexchange.com/questions/9425/bonferroni-or-tukey-when-does-the-number-of-group-means-become-large

Dear Ruth

What was the degree of freedom of ANOVA?

What was the df for unpaired t-test?

Have you used SPSS?

If you want to compare the mean of a control group with the mean of each treatment group, use Dunnett's Test. The experimentwise error rate will be smaller than that of Tukey's Test since the latter's error rate covers all pairwise comparisons. This results in a very small comparisonise error rate, leading to a lower power. Dunnett's Test is designed to only cover the k-1 pairwise comparisons whereas Tukey's Test covers all k (k-1)/2 pairwise comparisons.

Any multiple comparison procedure that controls the experimentwise error rate can be used without first having used the ANOVA F Test. Tukey's Test accounts for all possible pairwise tests. In your case, this is not what you are seeking according to your description of the problem.

A popular approach is using a Bonferroni or a Sidak Test in which multiple pairwise t tests are used. Tukey's Test is more powerful. Use Dunnett's Test for comparing control to treatment group. It can also be used with one tailed alternatives.

If you want to test everything against the control, you should use Dunnett's comparison against a control. Tukey compares every groups to every other group. This produces a large FWER and EWER.

Dunnett's test will only make 3 comparisons.

If the data are non-normal and you want to use Dunnett's test, consider transforming the data to normal scores first.

Hello everybody

I have performed an experiment where I treated cancer cells and measured certain cell damage in 3 different time points + untreated control. I have 4 replicates. I am thinking to use repeated measures one-way ANOVA followed by some post hoc test (Tukey or ?). Can you validate my thinking or correct it with providing an alternative?

Thank you

After the Anova, I would go with a Dunnett's test since you will compare each group to the control.