for KW you can do as you used to by using Wilcoxon–Mann–Whitney test followed by manual adjustment for p value by bonferoni method or you can use schefe in ANOVA as it based on ranks (i mean while doing the analysis by KW for non parametric data you can do post hoc using schefe in ANOVA procedure)

Hi Malgorzata,

There are at least three different approaches to this problem:

1) you can use Mann-Whitney test (for KW) and Wilcoxon test (for Friedman) and then adjust computed p-values. Once again there are many options: Bonferroni, Sidak, Holm (Bonferroni/Sidak) – in GraphPad Prism help it is explain how it works: http://www.graphpad.com/guides/prism/6/statistics/index.htm?stat_holms_multiple_comparison_test.htm

and there is of course False discovery rate (FDR) correction and modifications of FDR.

2) you can use rank transformation and then analyse data with ANOVA and “classical” post-hoc tests. See attached file for example, how to compute ranks for KW and Friedman test.

3) there are some nonparametric post-hoc tests, which however are mainly modified “classical” post-hoc tests that work with ranks. See: http://www.amazon.com/Practical-Nonparametric-Statistics-3rd-Conover/dp/0471160687

http://www.amazon.com/Nonparametric-Statistical-Methods-Probability-Statistics/dp/0471190454

Attachment

Dear Eman and Peter,

thank you very much for your answers and suggestions!

I've already used Wilcoxon test with p-adjustments.

I will try some data transformation as for ANOVA power computations are easier, at least in R statistical language.

Dear Malgorzata,

There are other alternatives that I would think are better than those already suggested. They are better for several reasons but primarily because they retain more information than any ranking methods does.

First, unless your data error structures are extremely odd you should be able to analyse it with a generalized linear model with appropriate link functions. This would be the most effective approach.

Second, you can simulate the test distribution by repeatedly randomizing the data. Each time you randomize you calculate the F. Then you find the percentage of repetitions in which the calculated value of F exceeded the F obtained from the original data. This is the p-value under the null hypothesis.

By the way, ranking data does not *necessarily* make it suitable for analysis by anova (or any other general or generalized linear model). You still must test the data to show that it meets the anova requirements .

Also note that the Wilcoxon and Mann-Whitney tests are exactly the same. Frank Wilcoxon published in 1945 and Henry Mann and Don Whitney in 1947. There is nothing to be gained or lost by applying one to KS and the other to Friedman tests.

best, Andrew

I used the Nemenyi test for this...it can be calculated in R or SAS

Note though that the Nemenyi test is valid only when sample sizes are equal (i.e. when you have the same number of replicates in each of your categories). If this is not true then Dunn's multiple comparison test is correct.

These particular post-hoc tests are modifications of the Mann-Whitney test approximation for large samples and so, likewise, they assume that sample sizes (number of replicates per category) are large.

Dear Andrew,

Sounds interesting. I've never thought before about GLM. It seems as really worth trying procedure.

Thanks for the suggestion!

Dear Peter,

Thanks for the suggesting Nemenyi test!

You cannot use the Wilcoxon or U Mann-Whitney statistics as post hocs for yor nonparametric Kruskal Wallis and/or Friedman ANOVA. There are specific nonparametric post hocs basing on the Tuckey statistics. You may find it in D.Heath 'An Introduction to Experimental Design and Statistics for Biology' 2000, CRC Press, page 280. or in J.H.Zar 'Biostatistical analysis' 2010, 5th edition, Pearson Education Int., pp.241-242. The latter is better because it takes into consideration the samples with either equal or unequal sizes. If you have further questions please write to my e-mail [email protected] so I can give you more indications.

Thanks you Joanna, I will try to find a book and a test you mentioned.

If you have 3 or more groups and suspect that a trend is at hand in your results you can calculate Jonkheere's trend test after Kruskal Wallis analyis of variance and after Friedmans test for related groups you can calculate  Page's L-test for a trend.

If you work in R, I've found pSDCFlig (Dwass, Steel, Critchlow, Fligner test) in package NSM3 to be a good option.

Holm-Sidak method can handle Multiple Comparisons versus Control Group and pairwise. It is more powerful than Tukey and Bonferroni.  IF you want to run a less conservative test use Student-Newman-Keuls.

I see here many different advices for different modeling approaches and also different tests to be used. Is there some theoretical foundation in place by which we can actually see which suggested approach should work better, or is each person recommending their own favorite choices, based on their own life experiences with data analysis?

Mathematics is the "theoretical foundation in place".

It's not a question of "works better" but which approach is most appropriate for the data (that is, which approach is least affected by factors like distribution shape and error structure)

Several factors lead to the great variety of advice.  Among these several factors are - lack of match with the question (i.e. the question is about post hoc tests, not trend tests), imprecision (e.g. saying a test has 'more power' without specifying under which conditions such a test is applied), and simply not knowing things.

best, Andrew

Dear Raid Amin,

My recommendations are based on my favorites and that they can be calculated by hand. We did not always have SPSS. Kruskal-wallis and Friedmann's are non-parametric tests, so you cannot recommend parametric tests as post-hoc tests.

For the Kruskal-Wallis test there are 2 different possible post-hoc tests, based on the critical difference of mean ranks. 1. a post-hoc critical difference after Conover, 2. a post-hoc critical difference after Schaich and Hamerle. The test after Conover is more liberal, whereas the thest after Schaich and Hamerle is exact but lacks a bit of power.

For the Friedman test there's also a post-hoc test comparing the critical difference of mean ranks.

You find an explanation on brightstsat.com and both tests are also implemented in brightstat's webapp:

https://secure.brightstat.com/index.php?p=c&d=1&c=2&i=7

https://secure.brightstat.com/index.php?p=c&d=1&c=2&i=9

Dunn test. Do not use Mann-Whitney if you can help it. You may want to have a look at for the reasons:

Answer 11 at http://stats.stackexchange.com/questions/111869/can-mann-whitney-test-be-used-for-post-hoc-comparisons-after-kruskal-wallis/111891#111891

or analogously Answer 9 at http://stats.stackexchange.com/questions/25815/difference-between-post-hoc-test-after-kruskal-wallis-and-bonferroni-corrected-m/95270#95270

...both are answers by guy who seems to know what he is talking about.

I have also seen sometimes recommendations for Nemenyi. (http://cran.r-project.org/web/packages/PMCMR/vignettes/PMCMR.pdf )

I'm certainly not an statistician, so these recommendations are based on personal experience. Having said that certain tests are reasonably robust to deviations from assumptions. And the medical community (I'm not a clinician neither) seems reasonably happy with MW-U as post-hoc option for KW; to a extent contradicting my opening statement.

Final word, a leading statistician as Prof. Bland in his introductory book suggest we are all making too much about the infamous p-values.

Hope this help.

Cheers!

I have been a statistics professor for 28 years. My recommendation is to use normal scores testing. 

It can be calculated using Dunn's nonparametric comparison for post hoc KW testing and also with SPSS 19 and higher versions. It is very simple.

Look this: http://www-01.ibm.com/support/docview.wss?uid=swg21479073

There is an exact post-hoc test for Kruskal Wallis after Schaich and Hamerle or post-hoc test after Conover

Both tests are illustrated on brightstat.com and brightstat's webapp lets you compute the critical differences and perform the posth-hoc analysis right away.

https://secure.brightstat.com/index.php?p=c&d=1&c=2&i=7

 Mann-Whitney or Wilcoxon tests with a Bonferoni correction. The latter if n is smaller than 6. 

These links may help you

https://www.researchgate.net/post/Which_post_hoc_test_is_best_to_use_after_Kruskal_Wallis_test

https://cran.r-project.org/web/packages/PMCMR/vignettes/PMCMR.pdf

https://stats.stackexchange.com/questions/175441/friedmans-test-is-very-significant-but-its-post-hoc-comparisons-spss-are-not

http://www-01.ibm.com/support/docview.wss?uid=swg21477370

https://www.researchgate.net/post/what_is_the_post_hoc_test_for_Friedman_test

https://www.r-statistics.com/2010/02/post-hoc-analysis-for-friedmans-test-r-code/

https://stats.stackexchange.com/questions/10309/friedmans-test-and-post-hoc-analysis

http://www-01.ibm.com/support/docview.wss?uid=swg21508972

you can read this

http://www.tandfonline.com/doi/full/10.1080/03610918.2014.931971

I have sample size of four and i am comparing four algorithms. Is it needed to use Friedman test.?

Yes it is with 4 dependent observations in a group.

Thanks

Hi There Folks, the discussion was excellent on the topic! I would like to know how to do Kruskal-Wallis test corrected for some variables across groups, such as age and gender? as it is ANCOVA.

https://www.healthknowledge.org.uk/public-health-textbook/research-methods/1b-statistical-methods/parametric-nonparametric-tests

go here

Hallo Dejan, Use Generalized Estimation Equations with appropriate covariates. If the variables you have do not match any of the standard distributions or the distributions differ strongly between variables, use permutation techniques.

@Andrew Davis. yes GEE, but fur from my knowledge! Will look for it!

How to perform Nemenyi test?

SPSS Statistics added the Dunn or Dunn-Bonferroni post hoc method following a significant Kruskal-Wallis test In Version 19 and later Analyze>Nonparametric Tests>Independent Samples (not Legacy Dialogs) Specify (click the Fields Tab) to add dependent variables and a grouping factor, pairwise comparisons (click the Setting tab) using the Dunn-Bonferroni approach are automatically produced for dependent variables for which the Kruskal-Wallis test is significant. In the Output viewer, double click the Hypothesis Test Summary to open the Model Viewer output. Look in the lower right of the screen for the View drop-down menu and select Pairwise Comparisons to see the post-hoc test results. Note: SPSS performs Bonferroni adjustment by multiplying the Dunn’s P value by the number of comparisons.

Hi all, I hope it's not too late for an additional question on that topic: What if my Kruskal-Wallis was significant, but when I then take the post-hoc Dunn test with Bonferroni adjustment, only some of the unadjusted p-values are significant? None of the adjusted p-values is significant anymore. How do I interpret this? Thank you so much!

https://youtu.be/xLAUs4hL5Vs

Very easy, u can select case and select 2 group other make it deselecte. U can see above link of youtube.

For the Kruskal-Wallis (KW) test, the Bergmann-Hommels post-hoc test is one option.

it's still a puzzle to me why anyone is using Kruskal-Wallis in the first place when much better techniques are available

Would a M-W be suitable for paired comparisons when the K-W showed a significant difference? I'm not sure if this would need to be corrected for multiple comparisons though (which is the point of post hot tests in parametric analyses such as ANOVA)?

Yes, Debbie, you'd have to correct for multiple comparisons. However, unless your data has very odd distributions, there are more appropriate techniques than KW, JT etc. (with these, as with all rank based tests, you throw away a great deal of information.) Among these more appropriate techniques, please examine Generalized Linear Models, and Generalized Estimation Equation models.

And then, with very odd and divergent distributions, use permutation and randomization.

For Friedmann I like to use a Nemenyi-Test for post hoc analysis. It is a pair-by-pair comparison. Both can be easily done with R.:)

Sandra Zilker Could you kindly suggest a tutorial for Nemenyi test?

Both Kruskal-Wallis ANOVA and Friedman ANOVA tests are non-parametric versions or equivalents to the One-way Between Subjects ANOVA (Independent groups design) and the One-way Within Subjects ANOVA (Within Subjects design) tests respectively. Both tests are also extensions of the Mann-Whitney (Wilcoxon Rank-Sum) U and the Sign tests for two independent and dependent groups respectively.

Like the regular One-way Between Subjects ANOVA (Independent groups design) and the One-way Within Subjects ANOVA (Within Subjects design), the Kruskal-Wallis and the Friedman tests may show a statistically significant difference, but will not tell you where the difference lies.

To accomplish this, we need to apply pairwise comparison of the groups median (or mean) ranks:

(a) For K-W ANOVA: We use the Mann-Whitney U (Wilcoxon Rank-Sum) test for the Post-hoc pairwise comparison of groups median or mean ranks (e.g. groups 1 & 2, groups 1 & 3 and groups 2 & 3) if there are three (3) groups for example. However, this test is associated with inflation of the type 1 error rates (alpha risk). Correction of this problem required the use Bonferoni correction of the significance level (alpha) given by:

Bonferoni correction = alpha/Number of groups for pairwise comparison

This gives our new alpha level of significance for comparison with the calculated test statistic. This is the test I prefer to use and readily available and hands-on in SPSS.

Other Post-hoc tests for pairwise comparison include:

(i) Conover-Iman test

(ii) Nemenyi test

(iii) Bergmann-Hommels test etc

(b) For Friedman test: We use the non-parametric Wilcoxon Signed-Rank test (for dependent groups) for the Post-hoc pairwise comparison of groups median or mean ranks (depending on the shape of the data distribution) and again use the Bonferoni correction to avoid type 1 error rates inflation.

Yes, Bonferoni correction is conservative, however, we should remember that the initial tests we conducted have already violated the assumptions for parametric tests. Therefore, the conditions or issues of CONSERVATION should be kept aside as this has limited or no effect on non-parametric tests.

Please don't call Kruskal-Wallis or Friedman "anova". Neither of them are anovas because they don't analyse or partition variance. They are not equivalents of anova either because they test a different thing, medians and not means.

In any case, because better solutions are available, there's little need to use them these days. If your data and its errors are so oddly distributed that you can't use generalized linear models with appropriate link functions, or else GEE, then use permutation methods. All are better than rank tests.

ANOVA by ranks

If the shape of your distributions are similar, use the median ranks, but if they have different shapes, use the mean ranks my friend.

Vesna Gulin Beljak Sorry for my late reply. I have used library("PMCMR") in R. Please see the link for the functions included: https://www.rdocumentation.org/packages/PMCMR/versions/4.3 Let me know, if you have further questions and I am happy to help :)

Does anyone know any library in python for these tests and what are the best options?

Diego Raffo scikit-posthocs https://scikit-posthocs.readthedocs.io/en/latest/. Best applied on crosstabs after using pandas.melt() on them.

With non-parametric data and more than two groups, Dunn’s test can be used as a post-hoc. For SPSS users, Dunn’s test is built in, but for SAS users it is requested through by a Macro.

However, when data lack the assumption of normality, the Generalized Linear Mixed Models are now widely used. You have to choose the appropriate distribution and link function. For example, non-normal right-skewed data can be analyzed with the Gamma regression, and count data with Poisson regression.

The following is an excellent tutorial on the benefits of the Generalized Linear Mixed Models.

https://www.youtube.com/watch?v=nkkv96teOxs&t=389s

I agree with Mahmoud Elmaghraby.

Dear Dr. @Mahmoud Elmaghraby...

Dunn test not provides the answer of interaction term. so please tell me if there any post hoc test for non parametric tests that also gives the answer of interaction term.