Non-parametric analysis - Kruskal-Wallis Test

When the data is ordinally scaled or when you violate the normality and/or homogeneity assumptions it is advisable to use the Kruskal-Wallis statistic to test the null hypothesis that the two or more medians are equal.

Assumptions for the Kruskal-Wallis procedure:

Sample randomly selected. Examine whether or not you have met this assumption by scrutinize sampling procedure.

Dependent variable at least ordinally scaled. Examine whether or not you have met this assumption by checking to see that the dependent variable meets the definition of an ordinally scaled variable.

Underlying continuity.

 ......... See below links

http://www.theanalysisfactor.com/calculate-effect-size/

http://www.slideshare.net/SariCamarena/kruskal-wallis-test

For the Kruskal-Wallis H-test you can estimate the effect size w² (it's w, not omega) by dividing the H-value through N. w² is the effect size of the chi²-test and it works, because the H-values are chi² distributed. The problem is that w² has a range from zero to infinite, so it is difficult to interpret.

If you want to calculate eta² you have to treat the results of the H-test like an univariate ANOVA, but this produces only an imprecise estimate.

Reneh, the effect size for Kruskal-Wallis is the epsilon sqare. Here is the formula: H / [(n2 - 1) / (n+1)], where H is the K-W test statistic value, and n is the total number of observations. Please note that n2 in the above formula means "n squared". Good luck! :)

For everyone who isn't familiar with Python language, but is not scared to download for example pycharm and use program written by someone else to make calculations easier, here I give you formula of epsilon sqare:

def epsilon2(h, n):

    return h/((n**2 - 1)/(n+1))

print(epsilon2(h, n))

In command print, in place of 'h' an 'n' put simply:

h - K-W test statistic value

n - total number of observations

Then run program :)

I can help you in Spss

If you have Chi-Square than effect size=(Chi-Square)/(N-1

this effect size is known as eta squared just like in ANOVA

Yes you can get the chi square by using the legacy procedure in SPSS. 

I have the same question as Reneh Karamians. In R, you can get chi-square value from kruskal-wallis test. And there a pwr package in R you can do the power-analysis. However, there is no way for Kruskal-wallis power test in this package. I have use one-way ANOVA for my sample test. And here is the plot. Blue line is the sample size when power=0.8. i don't know if i can do this? Does anyone have better idea or I am totally wrong in this case.

Thanks!!!

Hey Reneh,

if you're running Kruskal-Wallis with only two groups, you can calculate

r=√(chi²/N) according to Rosenthal & DiMatteo (2001, doi: 10.1146/annurev.psych.52.1.59). This is equivalent to calculating r=Z/√N from Mann-Whitney U test results. Eta squared can be calculated as η²=r²=chi²/N. Note that the Kruskal-Wallis H test statistic is approximately chi²-distributed.

If you're conducting Kruskal-Wallis with three or more groups (k), which produces chi² values with k-1 degrees of freedom, you could first transform chi² into an F value with k-1 numerator degrees of freedom (dfn) and N-k denominator degrees of freedom (dfd; modified from Murphy & Myors, 2014, Appendix A, ISBN-13 978-1848725881):

F(dfn,dfd)=chi²/(k-1)

Second, from F(dfn,dfd) you can calculate partial eta squared according to Cohen (1965; see Lakens 2013, doi: 10.3389/fpsyg.2013.00863):

ηp²=(F x dfn)/(F x dfn + dfd)

Because the Kruskal-Wallis test statistic is based on a single factor entered into analysis and consequently, no other variable accounts for variance in the dependent variable, ηp² equals η².

Example – 120 subjects, 3 groups, and Kruskal-Wallis chi²=7.5 (p=0.023518):

1) F(3-1,120-3)=7.5/(3-1)=3.75

2) η²=(3.75 x (3-1))/(3.75 x (3-1) + (120-3))=0.060

Note that the F value 3.75 as derived from step 1 is an approximation. The exact value of F(dfn,dfd) can be determined using the following R command with the Kruskal-Wallis p-value:

qf(1-p,dfn,dfd)

Thus, in our example, you may run the following code in R:

qf(1-0.023518,3-1,120-3)

R reveals an F value of 3.87. Again you calculate η² according to step 2:

η²=(3.87 x (3-1))/(3.87 x (3-1) + (120-3))=0.062

Best regards!

There is an on-line calculator:

https://www.psychometrica.de/effect_size.html#nonparametric

The psychometrica on-line calculator and Philippe Jawinski's method lead to different results. Does anyone have an idea why this is the case?

Thanks! Best, Michael

Hi Michael,

the psychometrica online calculator uses the formula provided by Barry Cohen (2008, see also the review article by Tomczak & Tomczak 2014: "The need to report effect size estimates revisited"):

eta²H = (H – k +1) / (n – k)

where k is the number of groups and n represents the number of total observations.

Barry Cohen's formula appears to reveal slightly smaller effect sizes when compared to the procedure that I suggested above. In fact, his formula even produces negative values if H + 1 is smaller than k. The latter is the reason why I was looking for a different approach. Nevertheless, Barry Cohen is a solid source to cite and I have no reservations to recommend this formula. In the end, it should be kept in mind that both procedures only enable to calculate nonparametric estimates of eta squared.

Best, Philippe

Many thanks for that quick and useful answer, Philippe. It is appreciated.

I am not sure, though, what you want to say with your last sentence. What else would you like to estimate? Apart from that, eta squared can be converted into Cohen's d if I am not mistaken.

Hi Michael,

I'm sorry for this ambiguous sentence. I simply intended to emphasize that both methods produce an approximate estimate of eta squared that is based on ranks instead of the observations themselves. While nonparametric tests have several advantages, one disadvantage is the loss of information about exact magnitudes. Regarding your second comment: Of course, the eta squared estimate may subsequently be converted into Cohen's d, which then also refers to ranked data.

Best, Philippe

It seems to me that epsilon-squared ---- as described by @AnnaKamza ---- is commonly mentioned as an effect size for the Kruskal-Wallis test.

There is another statistic called Freeman's theta that you'll see listed sometimes.

I have to say that I find calling an effect size for Kruskal-Wallis "eta-squared" to be odd, since eta-squared is based on the variance or sum of squares. I understand that authors are using the eta-squared for Kruskal-Wallis to be analogous to eta-squared for ANOVA, but this seems like a poor, and probably confusing, analogy.

I need to read up on Cohen's explanation of eta-squared for Kruskal-Wallis. I also wonder how this statistic compares with epsilon-squared in different cases.

Thank Salvatore. Does anyone know a good reference for epsilon2 including some hints on how to interpret the effect size? I didn't find anything satisfying. Can epsilon2 be converted into Cohen's d?

On epsilon-squared:

The Tomczak and Tomczak article cites a textbook by King and Minium (Statistical Reasoning in Psychology and Education). I'll try to get my hands on that. I've been working a lot lately on gathering information on effect sizes for some tests, but it's sometimes difficult to find clear information.

I played a little bit with eta-squared and epsilon-squared. EDIT: EDIT 2: For any given value of k, eta-squared is linearly related to epsilon-squared. However, this relationship varies as k varies. By my way of thinking, either epsilon-squared or Freeman's theta is more stable in interpretation than eta-squared across values of k.

You can see my suggestions for interpretation of epsilon-squared here, although they are subject to change: http://rcompanion.org/handbook/F_08.html

* * *

Since Cohen's d involves the difference in means and standard deviations, to me, it doesn't make sense to call anything based on ranks "Cohen's d". The nonparametric analogue is the r mentioned in the original question.

I made this file in excel, just replace the values ​​to get the square epsilon coefficient. It is in Spanish and English.

As a further note on these statistics, if there are two groups, Freeman's theta is the same as the absolute value of Cliff's delta, which is somewhat commonly mentioned as an effect size for Wilcoxon Mann-Whitney.

Thanks for all, very useful...!

Another approach: In their book on effect sizes, Grissom and Kim talk about "one-against-the-rest" and "one-against-all" effect sizes for designs with multiple groups. In the one-against-the-rest case --- say you have three groups A, B, and C --- to determine the effect size of A, you would make A one group and B and C the other and use an effect size for two groups such as Cliff's delta or Vargha and Delaney's A. In this way you get an effect size for each of groups, which may be more informative than a single effect size for all groups.

Perhaps someone can "unconfuse" me. In the July 8, 2017 posting by Philippe Jawinski for a two-group Kruskal-Wallis he demonstrates how to calculate r, which is *very* helpful. However, he then goes on to detail how to calculate eta-squared resulting in two nonequivalent measures of effect size for one statistical test. Would someone please explain to me 1) the differences in eta-squared and 2) the difference in *interpretation* in each effect size measure? Thank you very much.

Gregory E Gilbert , for more than two groups, the eta-squared value as described by Tomzcak and Tomzcak is the one that's usually used [ eta²H = (H – k +1) / (n – k) ]. It appears you can get negative values from this approach. It's really difficult to find any reliable interpretation criteria for these kinds of statistics. One issue to be aware of is that they may not be able to reach a value of 1. Another is that they don't have any natural interpretation. Also note that eta-squared and epsilon-squared will always be positive, so you'll never get a confidence interval that crosses zero. For me, I think a statistic like Vargha and Delaney's A makes more sense. First, it has a natural interpretation (the probability of a value from one group being larger than one from the other group). Second, its values can be on either side of 0.5 (or 0 for Cliff's delta), so you get meaningful confidence intervals for small effect sizes. VDA is used only for two samples, but looking at the statistic pairwise for groups, or reporting the maximum statistic from pairwise groups makes sense and preserves the common sense interpretation of the statistic.

Thank you so much Salvatore. Your contribution in this flatform has been always helpful. I also read about some statistical approach in your website but my problem is that am not good at using R.

Sal Mangiafico , is easy Freeman's Theta, for compute it in Ms Excel?

Cesar Merino , I doubt it would be easy in Excel, because you need to loop through the data or operate on vectors. On the contrary, you can use the R package rcompanion to calculate Freeman's theta. (Caveat: I am the author of this package). You can also request confidence intervals for this statistic. Note that Freeman's theta is applied to one ordinal variable and one nominal variable. In the function, you need to indicate which variable is the nominal variable. If your data is in table format, you can run the following here, or install R: https://rdrr.io/snippets/ .

Input =(

"Breakfast Never Rarely Sometimes Often Always

Travel

Walk 6 9 6 5 2

Bus 2 5 8 5 3

Drive 2 4 6 8 8

")

Tabla = as.table(read.ftable(textConnection(Input)))

Tabla

library(rcompanion)

freemanTheta(Tabla,

group = "row")

Sal Mangiafico , yes, I look for the web of the book, and several times I back. A lot of congratulations for your work. I will cited you in my next works. Thanks.

I'm looking for a reference that provides the meaning or value of the effect size in eta square for Kruskal-Wallis test. thank you

Kathryn Dorney Were you able to find any widely accepted threshold points for the effect size as measured by eta-squared?

There are interpretation ranges given here: https://rpkgs.datanovia.com/rstatix/reference/kruskal_effsize.html , but I suspect they were adopted from interpretation for eta-squared for anova: https://en.wikiversity.org/wiki/Eta-squared . EDIT: Yes, the values given by the last website are the range values given by Cohen (1988) for eta-squared used for anova. I don't know if it's fair to adopt these for Kruskal-Wallis. The book is Statistical Power Analysis for the Behavioral Sciences, 2nd. That's about as "authoritative" as it comes for the interpretation of effect size statistics, but remember that these interpretation values would differ by discipline, situation, and real-world consequences. They are ultimately arbitrary.

Sal Mangiafico thanks for the clarification.

Arka Ghosh , but also note that that's eta-squared for K-W, and your other question (https://www.researchgate.net/post/Is_there_a_definitive_source_for_interpreting_effect_sizes) was about epsilon-squared for K-W. They usually aren't terribly different.

Several articles and textbooks mentioned above likely made errors in formulating η² (eta-squared) and ε² (epsilon-squared) based on a Kruskal-Wallis Rank Sum Test with its chi-square statistic H. The formulae I consider correct and choose to follow are

• η² = H / (n - 1)
• ε² = (H - k + 1) / (n - k)

where k is the number of groups, and n is the total number of observations. The two formulae have been misidentified in publications and thus mismatched in statistical packages such as R packages of rcompanion up to version 2.4.1 by Sal Mangiafico and rstatix as of version 0.7.0.

In Tomczak & Tomczak (2014), formulae for η² and ε² are reversed because the source Cohen (2013, Chapter 21, p. 20, 34) misgave the formula for η². The chapter on "STATISTICAL TESTS FOR ORDINAL DATA" in Explaining psychological statistics https://higheredbcs.wiley.com/legacy/college/cohen/1118436601/pdf/Ch_21.pdf available not in a printed version but only on the publisher's student companion site for the book https://bcs.wiley.com/he-bcs/Books?action=index&itemId=1118436601&bcsId=8544 names a statistic "eta-squared" with a formula (Formula 21.8) plausibly intended for ε², causing the confusion. The formula for eta-squared as η² = H / (n - 1) is seen in a course handout for Kruskal-Wallis test http://oak.ucc.nau.edu/rh232/courses/eps625/handouts/nonparametric/the%20kruskal-wallis%20test.pdf and in a forum http://www.talkstats.com/threads/post-hoc-power-analysis-for-a-welch-anova-and-also-a-kruskal-wallis-test.70014/. The notes by Stikker (2017) https://drive.google.com/file/d/0B3VsxBCbBYAKb1kxNDZsWWJNcVE/view?resourcekey=0-mQ3G51TR_U_YHJm3slpjow shows clearly that the formula assigned to epsilon-squared in Tomczak & Tomczak (2014, p. 24) actual yields what is commonly known as an eta-squared quantity demonstrated in an SPSS example https://www.youtube.com/watch?v=Rna4VHHu4cI.

The difference between η² and ε² is easier to comprehend in ANOVA and linear regression models where the two effect sizes are commonly used. η², the proportion of variation in outcomes explained by predictors, equals the sum of squares of effects divided by total sum of squares in ANOVA https://www.theanalysisfactor.com/calculate-effect-size/, which is simply the popular R² well known in an OLS regression. The formula for Kruskal-Wallis's H given at https://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance readily shows that H / (n - 1) denotes the proportion of variation in rank transformation of outcomes explained by grouping factors, which is η² of the corresponding linear model and also equal to the square of Pearson correlation coefficient between the rank transformed ordinal variable and a dummy of two groups, so called rank biserial correlation. The connection between Kruskal-Wallis's H and F-test also indicates an underlying linear model (González Ariza et al., 2019). ε² on the other hand is a less biased estimator of effect sizes for a smaller sample size and a larger number of predictors, which penalizes the number of parameters to estimate. ε² is simply the adjusted R² in an OLS regression. Therefore, η² > ε² in real cases, just as R² > adjusted R². Then no difficulty there is to suspect that formulae given in Tomczak & Tomczak (2014) are mistaken where the reverse comparison holds. In fact, when I tested the results of current implementation of η² and ε² calculations for Kruskal-Wallis test effect sizes using survey data in R,

• library(rcompanion); epsilonSquared(x = outcome, g = group)

gives identical results with

• kruskal_effsize(outcome ~ group, data)

since both are developed from the same formula via Tomczak & Tomczak (2014) https://rpkgs.datanovia.com/rstatix/reference/kruskal_effsize.html which claims to measure ε². However, results of both functions coincided with

• summary(lm(rank(outcome) ~ group))$r.squared
• cor.test( ~ rank(outcome) + as.numeric(group))$estimate^2

which clearly reports η². On the other hand,

• library(rcompanion); ordinalEtaSquared(x = outcome, g = group)

led to statistics identical to

• summary(lm(rank(outcome) ~ group))$adj.r.squared

which is apparently an ε² measure. These comparisons suggest indeed that formulae for η² and ε² given in certain publications and adopted in several statistical programs and tutorials deserve a switch.

For effect size between two groups on an ordinal variable, the formula involving Wilcoxon effect size r mentioned by Philippe Jawinski that shows r² = η² = z² / N = chi-square / N as in several articles (González Ariza et al., 2019, p. 7; Fritz et al., 2012, p. 12; Tomczak & Tomczak, 2014, p. 23) can be misleading. The statistics r, η², z, N likely refer to the population parameters since a normal approximation of z-score is used, and possibly needs replacing with sample estimators such as n-1 for N in estimation. Since η² = H / (n - 1) as discussed above, an η² estimate is greater than an r² estimate but asymptoticly equal. When I tested wilcoxonR() implementation in rcompanion package using survey data, the numeric value of

• wilcoxonR(outcome ~ group, digits = 22)^2

is virtually the same as

• kruskal.test(outcome ~ group)$statistic / length(outcome)

which suggests that the square of z in a two-sample Wilcoxon test approximately equals chi-square (H) in a Kruskal-Wallis test between two groups, because r = z / sqrt(n) is used in wilcoxonR(). However, the numeric value of epsilonSquared() that actually reports η² exactly equals

• kruskal.test(outcome ~ group)$statistic / (length(outcome) - 1)

which is greater than wilcoxonR()^2. The difference between r² = z² / N and η² = H / (n - 1) I presume is also partially due to the normal approximation when calculating the z-score, since adjusting the sample size estimator from n to n-1 alone in wilcoxonR()^2 did not equalize the two quantities. The rank biserial correlation formula for Mann–Whitney U test described on Wikipedia https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test#Rank-biserial_correlation, however, generated a very different coefficient from the square root of r² and η² in my test example. Based on the relationship between η² and a corresponding Pearson correlation coefficient discussed above, this disparity is unclear to be by desired design or due to mistakes.

As a temporary resolution, I recommend using ordinalEtaSquared() or equivalently adjusted R² in an underlying rank transformed linear regression for ε² (epsilon-squared) to measure effect sizes between an ordinal variable and a nominal variable with two or more levels, since ε² corrects for bias in η² which can be estimated by epsilonSquared() when necessary. I would also call for a revisit and revise of articles and statistical programs that mistake formulae of such quantities.

References

Jiarui Tao , Thanks for adding this. It is valuable. I understand what you are saying, and I agree.

So, the sources for the formulae I have, for epsilon-squared and eta-squared in the Kruskal-Wallis test context are:

• King, B.M., P.J. Rosopa, and E.W. Minium. 2018. Statistical Reasoning in the Behavioral Sciences, 7th ed. Wiley. Or 6th ed (2010).
• Cohen, B.H. 2013. Explaining Psychological Statistics, 4th ed. Wiley. Or 3rd (2008).
• The Tomczak and Tomczak article is an easily accessible source of these formulae.

Article The need to report effect size estimates revisited. An overv...

I'm pretty reluctant to deviate from these texts for their definitions of epsilon-squared and eta-squared in the Kruskal-Wallis test context. They seem to be mostly well-accepted, even if they deviate from the definitions used in the anova case. As strange as that is, that may just be the way that it goes.

Note also that the Stikker document you cite simply has the approach "epsilon squared for a Kruskal-Wallis test is the same as the regular eta-squared."

For my functions and website, what I think I need to do is provide the references for my functions, and to define the formulae used. I might also add that these formulae are likely reversed from the anova case, and might add a method that calculates them directly from the rank-transformed anova.

eta-h-2 = (H-k+1)/(N-k)

k = number of groups

N = total sample size