Dear Peter and Thom,

I think you missed that Joseph-Omer is not testing mean differences. He wrote so, but he used Friedman/Wilcoxon and not ANOVA/t-tests. So we can only be sure that he is comparing rank distributions. Friedmann/Wilcoxon would only be equivalent to test mean differences when the distributions were unimodal symmetric with similar variances... but then ANOVA/t-tests would be the far better choice (more "power").

So Joseph-Omer's question is essentially: how can a series of Wilcoxon tests be used to control a family-wise error rate (FWER). Afaik, LSD and HSD don't apply here because there is no studentized test statistic, so Bonferroni/Holm/Hochberg/Benjamini/Yekutieli are the only ways to correct the family of p-values.

A further comment to Peter:

MCPs like Bonferroni, Holm, Tukey, Dunnett do control the FWER. There is no significant ANOVA required to achieve this. These tests work on their own, independent from any ANOVA. Only software usually calculate the pooled variance estimates used in these tests for ANOVAs and use these values later, so that these tests are usually offerd "after ANOVA" - but this is only a methodological issue and does not mean that these tests could be done only in conjunction with ANOVA or that they would only allowed being performed after a "significant" ANOVA.

The exception I know is Fisher's LSD. This method is to compare three means (not more!). Here it is shown that the combination of a "significant" ANOVA and the resulting 3 p-values of the pairwise t-tests (using the pooled variance) does control the FWER, without any further correction ("ANOVA-protected t-tests").

My comments to Joseph-Omer:

Your analysis is not about testing mean differences but about rank-distributions. I think if the distributions of your response were unimodal symmetric you would have used a normal linear model, and as you have just 3 groups Fisher's LSD would work (but you could also use Tukey or Bonferroni; for 3 tests there is not much a difference; for more tests LSD does not control the FWER and Tukey's method will clearly outperform the overly conservative Bonferroni method).

So it it likely that you test something that you did not want to test. You should be clear about that. If you use Friedmann/Wilcoxon then you should not talk about testing means.

The other question to consider is if controlling a FWER makes sense at all. Is it really a kind of a screening, where you could "collect false-positives" in a list of "significant candidates" (as you have just 3 groups I doubt that) or are your groups more like a set of "control A"/"conlrol B"/"treatment" or "control"/"treatment A"/"treatment A+B" or similar? In these cases I would not see what family of tests would require holding a FWER.

There are different schools of thought:

When you perform an Anova/Friedman and find a significant effect you can proceed to posthoc pairwise comparisons. 

1. Some perform Bonferroni corrections for multiple comparison corrections according to the number of tests performed. Bonferroni assumes independent test. For the present case, this is conservative. In my view, overly conservative and will inflate false negatives.

2. As the posthoc pairwise tests are from a finite sample, e.g. 10 tests when you have a set of 5 only. The Tukey test takes this into account and applies a slightly less drastic correction. In my view, this is the method of choice.

3. Some people, including me sometimes, argue that the significant ANOVA replaces an a priori hypothesis and an LSD correction, i.e. none, is sufficient.Given that you should not proceed to pairwise tests if the Anova does not produce a significant effect, this argument has some appeal.

Best, Peter

Peter - I'm not sure I agree with 2. or 3.

2. The Tukey HSD method is safe - protecting the complete null and all partial null hypothesis - but has two main limitations. First it is not particularly powerful. Second it is hard to generalise to unequal variance and within-subjects effects.

3. It is perfectly OK to proceed to pairwise tests with a non-significant main effect. However, it is important to use an appropriate procedure to control Type I error. Tukey's HSD is a good example. It does not assume a significant ANOVA main effect and requiring one would make it even more conservative.

I would suggest a general modified Bonferroni procedure such as Holm test or Hochberg test. The Hochberg correction in particular is among the most powerful techniques for controlling a set of tests with correlated test statistics (such as pairwise comparisons), is fairly easy to compute and does not require significance of the main effect in ANOVA.

Dear Peter and Thom,

I think you missed that Joseph-Omer is not testing mean differences. He wrote so, but he used Friedman/Wilcoxon and not ANOVA/t-tests. So we can only be sure that he is comparing rank distributions. Friedmann/Wilcoxon would only be equivalent to test mean differences when the distributions were unimodal symmetric with similar variances... but then ANOVA/t-tests would be the far better choice (more "power").

So Joseph-Omer's question is essentially: how can a series of Wilcoxon tests be used to control a family-wise error rate (FWER). Afaik, LSD and HSD don't apply here because there is no studentized test statistic, so Bonferroni/Holm/Hochberg/Benjamini/Yekutieli are the only ways to correct the family of p-values.

A further comment to Peter:

MCPs like Bonferroni, Holm, Tukey, Dunnett do control the FWER. There is no significant ANOVA required to achieve this. These tests work on their own, independent from any ANOVA. Only software usually calculate the pooled variance estimates used in these tests for ANOVAs and use these values later, so that these tests are usually offerd "after ANOVA" - but this is only a methodological issue and does not mean that these tests could be done only in conjunction with ANOVA or that they would only allowed being performed after a "significant" ANOVA.

The exception I know is Fisher's LSD. This method is to compare three means (not more!). Here it is shown that the combination of a "significant" ANOVA and the resulting 3 p-values of the pairwise t-tests (using the pooled variance) does control the FWER, without any further correction ("ANOVA-protected t-tests").

My comments to Joseph-Omer:

Your analysis is not about testing mean differences but about rank-distributions. I think if the distributions of your response were unimodal symmetric you would have used a normal linear model, and as you have just 3 groups Fisher's LSD would work (but you could also use Tukey or Bonferroni; for 3 tests there is not much a difference; for more tests LSD does not control the FWER and Tukey's method will clearly outperform the overly conservative Bonferroni method).

So it it likely that you test something that you did not want to test. You should be clear about that. If you use Friedmann/Wilcoxon then you should not talk about testing means.

The other question to consider is if controlling a FWER makes sense at all. Is it really a kind of a screening, where you could "collect false-positives" in a list of "significant candidates" (as you have just 3 groups I doubt that) or are your groups more like a set of "control A"/"conlrol B"/"treatment" or "control"/"treatment A"/"treatment A+B" or similar? In these cases I would not see what family of tests would require holding a FWER.

Dear Joseph-Omer, Thom and Jochen,

thanks a lot for your question and answers, good discussion. First of all, I limit my comment to the ANOVA complex (the conceptual aspects generalize, the detailed math is different). Thus, it is only part of Joseph-Omer's question and somewhat misplaced. But as we happen to be here, I hope it is ok, Joseph-Omer, when you host this discussion thread.

When you perform an experiment with let's say three conditions (for two conditions the ANOVA reduces to a t-test and the question is void) you might ask, is there any pairwise difference in means. That is, for sampling X in condition i element {1,2,3} as Xi~N(mean_i, Sigma) is there a significant difference between  any mean_i and mean_j. In this situation performing the Tukey test is perfectly fine and valid. Admitted.

When you perform an ANOVA you ask a slightly different question. That is, is Are the data Xi better described by being sampled from Xi~N(mean_i, sigma) than Xi~(mean, sigma). This small but important difference is only the individual condition/group mean in the former in contrast to the joint mean in the latter case. If the ANOVA gives a significant result, you established that the detailed model is better than the simple model.  Technically, it means that the additional variance explained by introducing condition/group-specific means is worth while the additional number of freedoms (n-1, here 2). Fine and mission accomplished.

If the ANOVA is significant, in my view you might want to better understand what is driving the increased benefit of the detailed model. You may perform posthoc tests, with e.g. LSG correction. The ANOVA, however, is the main statistical test.

If the ANOVA is not significant and then you continue with pairwise tests you have three problems. First, you are multiple testing the same data with tweaking the question. Tukey does not correct for that. Tukey corrects when you perform it directly. But doing an ANOVA, continuing after a non-significant result with a further test is also multiple comparisons! Second, under which conditions could the pairwise test be significant without the ANOVA being significant? The ANOVA tells us the

- First, you are multiple testing the same data with tweaking the question. Tukey does not correct for that. Tukey corrects when you perform it directly. But doing an ANOVA, continuing after a non-significant result with a further test is also multiple comparisons! Second, under which conditions could the pairwise test be significant without the ANOVA being significant? The ANOVA tells us the the additional

- Second, under which conditions could the pairwise test be significant without the ANOVA being significant? The ANOVA tells us the additional variance explained by the detailed model sum_i(mean_i - grand mean)^2 is not enough to make the 2 additional degrees of freedom worth while. The pairwise test considers only the sum_i, not over 1,2,3, but just e.g. 1,2. Thus, the variance explained by the t-test is strictly smaller than the variance explained by the detailed ANOVA model. It introduces, however, only one new degree of freedom. Thus the variance explained is larger by a factor somewhere between 1 and 2. On the other hand, this small benefit is diminished by the necessary multiple comparison corrections. It can be a small effect at best.

- Third, when performing the experiment you considered all three conditions relevant. Then you ask, do I need a detailed model for each condition, or is one enough (the ANOVA). The answer (ANOVA not significant) is one is enough. You continue with asking, yes, but e.g. condition 1 vs 2, ignoring condition 3. Here is a semantic subtlety. You do not ask is condition 1 different from 2,3 combined or condition 2 different from conditions 1,3 combined etc. Otherwise, the ANOVA would have been significant. Thus, you are ignoring data you recorded and planned to analyze.

For these reasons I'd maintain that when you perform an ANOVA with no significant result, a decent option is to stop the further analysis.

Best, Peter

@Peter,

"under which conditions could the pairwise test be significant without the ANOVA being significant?"

Here is an example: 3 groups with 5 values:

           [,1]      [,2]      [,3]

[1,]   -544     275    -654

[2,]   2340  1122    -157

[3,]   1255    252   -559

[4,]    220   3663   -786

[5,]   -250  1868       63

These values were generated by sampling from a normal distribution with µg and σ=1, the µg (group means) were also sampled from a normal distribution with µ=0 and σ=1. The values were multiplied by 1000 and round to integers (for easier reporting).

For alpha = 0.05, the ANOVA is "not significant" (p=0.05733), but Tukey's HSD gives one "significant" comparison between groups 2 and 3 (p = 0.0471130).

Since Tukey does in fact hold the FWER (without considering any ANOVA), doing Tukey's tests only after "significant" ANOVA will have a lower effective FWER than  alpha (so this procedure is more stringent than desired).

In a simulation with 10000 runs (k=3, n=5) with data sampled under the null, the observed frequency of rejections at alpha=0.05 was 0.05068 (ANOVA), 0.11658 (pair-wise t-tests using the pooled variance), and 0.05076 (Tukey). Considering the pair-wise t-tests only in the cases where the ANOVA null was rejected gave a rejection rate of 0.05068 (this is Fisher's LSD). Cosidering Tukey only after rejecting the ANOVA null gave a rrejection rate of 0.0465, what is not drastically too low, but clearly lower than the nominal 0.05 it should have. 

So maybe I can summarize that Tukey after ANOVA is logically wrong and leads in fact to lower-than-nominal rejection rates under the null. However, this effect is not severe. If the calculation is a problem, it is worth to first check the ANOVA and continue only with Tukey when the ANOVA is significant, as this would save work on the calculations. But, really, calculation power is not any limiting factor nowadays (at least in the vast majority of applications; this may not hold for "big data" analyses - but there massive testing is usually not a smart way to go anyway).

Dear Peter, Thom and Jochen,

Thank you for this very enlightening discussion. Indeed, I will probably compare rank distribution (median) and not means as I wrote in my initial question. I'm sorry for my mistake and the confusion it generated.

I'm expecting an ''non-normal distribution'' that is why I'm preparing my self for using Friedman analysis (instead of ANOVA). I will follow-up a small sample of patients (12-20 patients max)  from pre-intervention (median a), 3 months post-intervention (median b) and 6 months post-intervention (median c). I'm expecting that the severity of pain will not be normally distrubuted because of the small sample size and obviously they all treated for moderate to severe pain...

As I understand the intervention of Jochen, I need to correct for FWER (follow up of same patients), so can I conclude that I should use the Tukey method?

I was also  wondering, How I should estimate the effect size after Wilcoxon test?

Thank you very much

Dear Joseph-Omer,

Testing a rank distribution is not the same as testing a difference in medians. It would be equivalent to testing differences in medians if all the characteristics of the distributions in the groups are all equal except the location. Only in this case can the location be describes by a single numer, which can essentially be any quantile of the distribution (like, for instance, the 0.5-quantile = median). But I would make any bet that this is never the case for biological data. Usually there is a connection between the mean and the variance, so that the distributions can not have the same shape when they have different means. Bottomline: you should not state to test medians either!

These tests test the null hypothesis that there is no rank-difference between the groups. For the Wilcoxon test this can be formulated as Pr(A>B)=0.5, where A and B are two random values from the two groups.

Multiple testsing / controlling the FWER:

If you, say, screen several time points in the follow up and you actually just want to see if there is something going on at whatever time point, then you should adjust your p-values for multiple testing to avoid overestimating their significance. If the time-points refer to independent questions, like "pre-OP", "post-OP" and "5-year follow-up", then -to my opinion- a correction for multiple testing is not required, because the tests do not belong to a closed family (like in a screening).

Tukey's HSD is for testing means, what you don't do. There is not much you can do with nonparametric tests except Bonferroni or Holm.

Regarding the effect size:

https://stats.stackexchange.com/questions/133077/effect-size-to-wilcoxon-signed-rank-test

https://www.researchgate.net/post/What_is_the_appropriate_effect_size_calculation_for_Wilcoxon_signed_rank_test_related_samples

https://www.researchgate.net/post/Any_suggestions_on_the_effect_size_to_Wilcoxon_signed_rank_test

Dear Jochen,

Once again thank you very much for your time.

In resume, I think I should expect (depending on the data distribution) that I will have to perform a Friedman, than Wilcoxon and Holm/Bonferroni correction for my follow up study.

Thank your for the references on effect size estimations.

best regards,

The Hochberg correction is more powerful (and is a modified version of the Holm procedure) is also suitable for this context. It is trivial to implement using p values using the p.adjust() function in R.

I'm not sure Friedman is a necessarily a good idea (see link). It is an extension of the sign test rather than Wilcoxon and can have rather low power (as it ignores available ordinal information about magnitude of differences). You can just rank the data and use a repeated measures ANOVA (and see comments in the linked article).

Another option (which more-or-less solves the effect size issue as you can use mean or median differences) is to bootstrap.