Dunnett's test was designed to compare each of k-1 treatments to a common control condition, so use it.  It does not require a significant omnibus F-test. 

Also, don't worry about the omnibus F-test and k-1 'treatment v control' contrasts giving different results.  They are addressing different questions, so there's no reason to expect the same answers! 

HTH.

http://davidmlane.com/hyperstat/B112114.html

What is the biologically relevant question?

I think that you have or expect a dose-response-relationship, and as such should your data be analyzed. To my opinion it is more useful/sensible to estimate the EC50 of the cytokine treatment.

The comparison of multiple groups (concentrations) to a control ignores any such relationship and gives you not-so-sensible and surely less useful answers.

But to your "problem":

You made different analyses (that also use different subsets of your data) - so you should not expect identical answers. This is not a problem, this is rather normal. The problem seems to be why you did different analyses in the first place. This makes no sense when the rejection of a null hypothesis is your aim. For a hypothesis test the whole analysis must have been defined beforehand, so there is no option to "try different analyses". If you do so, then the meaning of the p-value(s) is lost. If you do significance tests, it might(!) be acceptable to try different analyses, but if you do so, then the "surprise-value" of the p-values has to be adjusted. I have no idea how this can be done in an optimal way; using a Bonferroni correction seems too conservative, but this is only a gut feeling.

The reason for your different results ia that LSD uses a pooled standard error from all data, whereas the eparate comparisons estimate the standard error only from the data of the groups that are compared. Since the estimation of the standard error is relatively imprecise/uncertain, it is highly recommended to use as much data as possible in order to get a more robust estimate. This means: you should better use LSD.

@ Bruce:

Also, don't worry about the omnibus F-test and k-1 'treatment v control' contrasts giving different results. They are addressing different questions, so there's no reason to expect the same answers!

That is generally correct, but just not in this particular case if the FWER should be controlled! The "protected LSD" requires a significant ANOVA to control the FWER, and this works only for k=3 groups (and not for 5 as stated in the question).The "unprotected LSD" does not control the FWER.* (See: Fundamental Statistics for the Behavioral Sciences, 8th edition, David C. Howell)

It still is questionable if controlling the FWER is desired at all, but this is a different question... If the FWER should be controlled, your hint for Dunnett's test is surely the best option. But again: controlling error-rates makes sense only if the error-rates are selected wisely, what would require to think and specify the relevant effect sizes, used tests and sample-sizes when planning the experiment. Asking such questions after the experiment demonstrates that a sensible control of error-rates is just not possible anyway.

@ both:

* The unprotected LSD (not controlling the FWER) is to be preferred over the separate comparisons, because the pooled standard errors are used which are more robust.**

**This, surely, if recommendable only under the assumption of variance homogeneity (as ANOVA is, too). If this assumption is unreasonable that the main question is: why this? - it is likely that the whole model is mis-specified.

Jochen, for comparison of each treatment to a common control, I recommended Dunnett's test, which is also discussed in Howell's excellent chapter on multiple comparison procedures.  From p. 374 of the 6th edition (emphasis added):

"In some experiments, the important comparisons are between one control treatment and each of several experimental treatments.  In this case, the most appropriate test is Dunnett's test.  This is more powerful (in this situation) than are any of the other tests we have discussed that seek to hold the familywise error rate at or below [alpha]."

I believe Howell is wrong in classifying Dunnett's test as post hoc (because the 'control' condition must be known ahead of time).  But he is correct in noting that unlike Fisher's LSD procedure, Dunnett's test does not require a significant omnibus F-test. 

Cheers,

Bruce

Bruce, we perfectly agree. Given the control of the FWER is wanted and multiple treatments are to be compared to a single control, Dunnett's test is the best option.

But Dunnett's test really is a post hoc test because the whole data set must be known before any of the comparisons can be performed. I can cite Wikipedia here:

In particular, the t-statistics are all derived from the same estimate of the error variance which is obtained by pooling the sums of squares for error across all (treatment and control) groups. The formal test statistic for Dunnett's test is either the largest in absolute value of these t-statistics (if a two-tailed test is required), or the most negative or most positive of the t-statistics (if a one-tailed test is required).

It is obvious that this is not possible to do when not all the data are known to all tests, so the tests are done after all the data is known. So if Howell classifies Dunnett as post hoc, I think he is correct (I don't have the book, just referring to your statement).

Fisher's LSD has the possibiliy to be performed "unprotected". This provides the advantage that a pooled standard error (SE) is used (making the comparisons more robust) but a FWER is not controlled. The unprotected LSD is just a series of t-tests but using the pooled SE.

Sidenote w.r.t. "(because the 'control' condition must be known ahead of time)": If the control group is not defined and multiple comparisons against the "best" group should be done, Hsu's MCB test is an alternative to Dunnett.

Jochen, obviously I agree that you have to have all of the data in hand to carry out the pairwise comparisons of each treatment with the control.  But I do not agree that this makes it post hoc. Suppose I have a study with k = 4 treatment groups, and plan in advance the following contrasts:  1 v [2,3,4]; 2 v [3,4]; 3 v 4.  I cannot carry out this set of contrasts unless I have all of the data in hand.  But I think you would agree with me that if they are planned in advance, they are planned (and orthogonal in this case) contrasts, and therefore are considered a priori. 

So, I'll see your Wikipedia excerpt, and raise you a quote from Glass & Hopkins (1996, p. 457, emphasis added).  ;-)

"The Dunnett MC method is tailor-made for the situation where the plan is to compare each of the J - 1 means with one, and only one, predesignated mean (usually the mean of the control group).  Thus with the Dunnett method, it is assumed that there are C = J - 1 planned pairwise contrasts--each contrast is against the mean of the predesignated control group, [Xbar_*]."

Glass & Hopkins go on to say that one could make the same J - 1 contrasts via Dunn's test (which Howell classifies as an a priori procedure).  But the critical value for Dunn's test is slightly higher than the critical value for Dunnett's test.  This is another part of the argument for classifying Dunnett's test as an a priori procedure:  It would make no sense for a post hoc procedure (i.e., Dunnett's test, according to Howell) to have a lower critical value than an acknowledged a priori procedure (Dunn's test) for the same contrasts.  One of the reasons to use a priori procedures when they are applicable is that they have more power. 

I wrote to Howell about this issue several years ago, but never persuaded him to change his mind about it. 

Cheers,

Bruce

References

Glass, G. V., & Hopkins, K. D. (1996). Statistical methods in psychology and education. Allyn & Bacon, Needham Heights, MA.

Howell, D. (2007). Statistical methods for psychology (6th Ed.). Thomson Wadsworth.

So this seems to be the key point of our disagreement:

- you think that post-hoc depends on the plans about which comparisons to make; if they are specified befor conducting the tests, the tests are not post hoc.

- I think that post-hoc depends on the fact that each comparison (planned or not) is made based on information from the whole set of data.

I have no (good) source at hand supporting either opinion. I have no source of a proper definition of "post hoc". However, most references I could find agree better with what you said. So likely I am wrong and you are right.

So given I am wrong, I have a problem with teh disctiction - you can probably help me here:

The control of the FWER seems to be clearly independent of "post-hoc". I can define a family for any group of tests or p-values, and I may wish to control P(at least one false rejection of H0 among this family) < alpha. Am I correct so far?

Now to the "planned comparisons": from my experience, if a researcher conducts an experiment with several groups, she wants to compare either all pairwise combinations or all comparisons to a control. Aren't these "planned comparisons"? These groups were chosen to compare them... so there are almost never any unplanned comparison (like "oh, look, groups B and G look really different, so le's compare them using a test). I do not see the point in distinguishing "planned" and "unplanned" comparisons. But what I think is really essential is if a pooled standard error is used.

The original question was about an experiment with 5 doses compared to a control. The aim of the whole experiment was to make these 5 comparisons. So they were "planned" - then suggesting any post-hoc test would be wrong.

For me this all makes little sense. For me there are two sensible questions: should or can a pooled standard error estimate be used (to get more robust results), and should a FWER be controlled? I do not see how the fact if comparisons were "planned" or not should define how to analyze the data. Maybe I miss something impotant. But so far I think that the "planned comparisons" was an unfortunate phrase used by some statistician that was not right on the point but has proliferated through literature. How do you think about this?

JW:  So this seems to be the key point of our disagreement:

- you think that post-hoc depends on the plans about which comparisons to make; if they are specified befor conducting the tests, the tests are not post hoc.

BW:  Yes, that's it, more or less.  Although I think I might have a special category for all pair-wise comparisons.  As you say below, the plan from the beginning may be to carry out all pair-wise comparisons, which makes them "planned" by my definition.  But if it only occurs to one after having examined the data to carry out all pair-wise comparisons, the same method would (likely) be used--Tukey's HSD would be a common choice.

JW:

- I think that post-hoc depends on the fact that each comparison (planned or not) is made based on information from the whole set of data.

I have no (good) source at hand supporting either opinion. I have no source of a proper definition of "post hoc". However, most references I could find agree better with what you said. So likely I am wrong and you are right.

So given I am wrong, I have a problem with teh disctiction - you can probably help me here:

The control of the FWER seems to be clearly independent of "post-hoc". I can define a family for any group of tests or p-values, and I may wish to control P(at least one false rejection of H0 among this family) < alpha. Am I correct so far?

BW:  In my notes on multiple comparison procedures, here's what I say about the distinction between a priori & post hoc MC methods.

1.11  Planned versus Post Hoc Comparisons

            There is a very important distinction between planned (or a priori) and post hoc (or a posteriori) contrasts.  Glass and Hopkins (1984, p. 380) say this about the distinction:

"In planned contrasts, the hypotheses (contrasts) to be tested must be specified prior to data collection.  MC methods which employ planned comparisons can be advantageous if the questions that the researcher is interested in are a relatively small subset of questions.  The distribution theory and probability statements for these MC methods are valid only if there is no chance for the user to be influenced by the data in the choice of which hypotheses are to be tested.  The rationale for planned contrasts is similar to that for “one-tailed” t-tests--to be valid, the decision must be made a priori.  Post hoc MC techniques do not require advance specification of the hypotheses (contrasts) to be tested.  The Tukey and NK MC methods are considered to be post hoc methods since there is no delimitation as to which pairs of means will be contrasted."

            To further illustrate how important this distinction really is, consider the scenario described by Howell (2007, pp. 345-346).  If you have 5 means, there will be 10 possible pair-wise comparisons.  Let us imagine that the complete null hypothesis is true (i.e., all 5 population means are equal).  Let us also imagine that the smallest and largest sample means differ by enough to allow rejection of the null hypothesis for a pair-wise comparison, but that no other pair-wise comparisons would allow rejection of the null hypothesis.  Now imagine that you are going to carry out one pair-wise contrast.  As Howell points out:

"If you have to plan your single comparison in advance, you have a probability of .10 of hitting the 1 comparison out of 10 that will involve a Type I error.  If you look at the data first, however, you are certain to make a Type I error, assuming that you are not so dim that you test anything other than the largest difference."

JW:

Now to the "planned comparisons": from my experience, if a researcher conducts an experiment with several groups, she wants to compare either all pairwise combinations or all comparisons to a control. Aren't these "planned comparisons"? These groups were chosen to compare them... so there are almost never any unplanned comparison (like "oh, look, groups B and G look really different, so le's compare them using a test). I do not see the point in distinguishing "planned" and "unplanned" comparisons. But what I think is really essential is if a pooled standard error is used.

BW:  I guess I've always bought what Glass & Hopkins say above about all pair-wise contrasts being treated as post hoc, because "there is no delimitation as to which pairs of means will be contrasted". 

JW:

The original question was about an experiment with 5 doses compared to a control. The aim of the whole experiment was to make these 5 comparisons. So they were "planned" - then suggesting any post-hoc test would be wrong.

BW:  I agree.  And as Dunnett's test is the clear choice for this situation, it should not be classified as a post hoc test.  ;-)

JW:  For me this all makes little sense. For me there are two sensible questions: should or can a pooled standard error estimate be used (to get more robust results), and should a FWER be controlled? I do not see how the fact if comparisons were "planned" or not should define how to analyze the data. Maybe I miss something impotant. But so far I think that the "planned comparisons" was an unfortunate phrase used by some statistician that was not right on the point but has proliferated through literature. How do you think about this?

BW:  I agree with your two questions, but would add a third, which is how can FWER be controlled while maximizing power?  As Howell's example above shows, if the single contrast must be specified in advance (i.e., before seeing the data), the probability of Type I error is 1/10, vs 10/10 when the choice of means to compare is made after seeing the data. 

Finally, I think that we are probably mostly in agreement (as usual).  The main point I am trying to make in classifying Dunnett's test as an a priori procedure is that it would not be fair to wait until one has seen the results, and then choose one of the means to serve as the "control" mean.  In my view, the p-values from Dunnett's test are only valid if the control group & treatment groups are known a priori.  If it really was a post hoc procedure, on the other hand, then one could validly choose the 'control' group after having seen the data.  That's how I see the distinction between a priori & post hoc in the context of Dunnett's test.

Cheers,

Bruce

Thank you Bruce for the detailed answer. I have to digest this :)

For now, I think that a sensible specification of a desired FWER requires to denote some "relevant effect size", so the whole experiment needs to be planned anyway. This means that - if the FWER is to be controlled - I automatically have to have a completely planned experiment, I have to have determined the requeired sample size for the well-specified groups and comparisons. Thinking in this line any tests that aim to control the FWER cann in principle be no post-hoc tests. But Tukey's HSD, NK and many others that are designed explicitely to control the FWER are commonly termed "post-hoc tests". This I do not understand. Ok, one could argue that a FWER can be controlled technically at any level, but: 1) controlling only type-I errors without controlling type-II errors is practically non-sensical and 2) without having a particular relevant effect size and a power in mind there is no sensible justification of a particular FWER possible (it's all arbitrary, and it is not possible to judge a practical benefit from the whole procedure*)

* many people say that "not making too many false-rejections is the main objective, and this is controlled", but it may be worse to miss a  correct rejection. Apart from this, we run into the usual problems when we reject point-null hypotheses where a relevant alternative is not specified. No relevant alternative -> no sensible justification of an "inductive behaviour" -> the whole fundament of hypothesis tests is eroded.

Jochen (and others who are following this thread), I just found a link for an "old version" of Dave Howell's nice chapter on multiple comparison procedures in his book Statistical Methods for Psychology (see below).  Here's what he says about it:

"In the text I said that I would provide a copy of an older chapter that covers a wider variety of multiple comparison techniques in greater detail. If you really want that stuff, here it is. Much of the chapter is similar to the one in the 8th edition, but there is a lot that has been left out of the new version."

HTH.

http://www.uvm.edu/~dhowell/methods8/Supplements/OldChapter12.pdf

http://www.uvm.edu/~dhowell/methods8/Supplements/SuppIndex.html