Mann-Whitney U test. Odd that your groups are not normally distributed when they have such a large sample size. Have you checked with Kolmogorov-Smirnov's test?

I am trying to think of a hypothesis that would require pairwise testing of 10 to 30 groups!

First, note that the Friedman test is a pretty low power test, and is not, as people think, an extension of the Wilcoxon Mann-Whitney test – it's a sort of sign test max. It also tests a hypothesis that is pretty hard to define! Thom Baguley has some useful remarks here : https://seriousstats.wordpress.com/2012/02/14/friedman/

He points out that you are better off doing and anova on ranks than going with the Friedman.

However, for ten groups you have 45 possible pairwise comparisons, while for thirty you have over four hundred. Step back a bit and ask yourself if these are hypothesis tests. My general principle is no hypothesis – no test.

If the groups can be simplified by merging categories that are a) similar to each other and b) different from the other categories, then you can reduce the number of groups, which will reduce the post hoc testing, but it still won't give you a hypothesis. If the groups were, for example, geographical districts you could pool them into regions that would be recognisable as meaningful entities and different one from the other. If they were diagnoses, you could group them by body system or by procedure type etc etc.

If, on the other hand, the groups are of actual interest, for example the grouping variable is a school, then multilevel modelling is more appropriate because you can then test hypotheses about the effect of context (group) on individual observations.

I would comment that Béatrice Marianne Ewalds-Kvist 's surprise that your outcome variable is not normally distributed assumes that it ought to have been. In real life there are many other distributions – if you are counting things, for example, Poission or negative binomial distributions make more sense, and many distributions are zero-inflated because the variable doesn't apply to some people – so I'm not convinced that you need to run from the data in Friedman's direction just yet.

Tell us more about your study and about your hypothesis, and critically about the outcome variable.

Yes, It was made. The data is not normally distributed.

Thank you all for the answers!

Finding a negative probability is exciting enough, you should just publish that! Or edit your question and as Ronán Michael Conroy says, say what your hypothesis is. Also say why you think the central limit theorem may not be that useful here (maybe saying a bit more about your data). It might also be worth examining multilevel (mixed) models before responding, as these may be applicable. But first publish that negative probability! ;o)

Mann-Whitney is not an appropriate post-hoc for the Friedman test. This test is equivalent to the Wilcoxon rank sum test.

There are few post-hoc tests that have been developed for the Friedman test, including those that address all pairwise comparisons, and those that compare one group to the rest. Common ones include the Conover test and the Nemenyi test. A good list can be found at the index here, those functions with names beginning with "frd". https://cran.r-project.org/web/packages/PMCMRplus/PMCMRplus.pdf

If your software has limited options, I suppose you could use pairwise two-sample paired sign tests. That's sort of similar to the way the Friedman test operates.

Since you have many groups, you might consider summarizing pairwise tests with a compact letter display.

P.S. I think the Friedman test is fine to use in appropriate situations. Particularly when you have truly ordinal data. I certainly don't think it's an imposter in any way.

Salvatore S. Mangiafico , on Friedman's, if I understand the question these are separate groups. Is this how you understand this? There are a few issues with this question, so hopefully the person can clarify stuff.

Béatrice Marianne Ewalds-Kvist , "Odd that your groups are not normally distributed when they have such a large sample size":

The distribution of a variable has nothing to do with the sample size. If your variable has a distribution D, than it has the distribution D. No matter how many values you sample. Larger samples sizes will only give a clearer picture that the distribution is from D (or from E or F...).*

The central limit theorem is about the distribution of a sum (or a mean) of a large number of random variables: no matter what the distributions of these random variables is (under some "mild regularity conditions"), the distribution of the sum (or mean) will approximate the normal distribution with increasing number of variables that are summed up (or averaged).

---

* This is one of the problems of "formal tests of assumptions": No real variable has a perfect normal distribution ("N"). If D is approximately N, it only takes a sufficiently large sample to show that D is not identical to N. And a sufficiently small sample will never suffice to show this, even when D is relevantly different from N. Hence, as always, the significance test is not about whether you can conlude D or N - it's rather about to conclude if your sample size is large enough to discern your unknown distribution from N.

Jawohl Jochen Wilhelm,

Sie haben immer recht, und es ist immer besser mit einer größeren Stichprobe

Gruss aus Finnland!

Ich wünschte es wäre so :)

Liebe Grüße aus Deutschland!

Most statisticians recommend conducting post hoc tests that are neither liberal nor conservative. Midway post hoc tests include the Tukey HSD test and Games–Howell test, depending on whether the assumption of homogeneity has been violated. Simply put, if Levene’s test were not statistically significant, the Tukey HSD test should be carried out. If it were significant, the Games–Howell test should be conducted.

That being pointed out, and since you have many groups, you should bear in mind that the statistical power of group comparisons decreases as the number of comparisons increases. Therefore, you should consider reducing the number of comparisons to alleviate the power tradeoff in post hoc tests. In this regard, you could check out Frost’s (2020) book, Hypothesis Testing, for relevant inputs.

Frost, J. (2020). Hypothesis testing: An intuitive guide for making data driven decisions. https://books.google.com/books/about/Hypothesis_Testing.html?id=2RnZzQEACAAJ

Good luck,

PS: Book cover is attached.

Thank you all very very much for your answers!

It will be noted.

I think that Dunn's and Tukey's tests are both a good decision for my goal. -), but not the Mann-Whitney....

Oleksiy Voytsekhivskyy, you said you have 10 to 30 groups with 200-500 observations in each. That description--especially the varying number of observations per group--suggests to me that you have 10 to 30 independent groups. If you do have independent groups, the Friedman test is not appropriate. Please clarify the nature of your data. What is the DV, and do you have independent groups, or repeated measures such that each subject has a score in each condition? Thanks.

Oleksiy Voytsekhivskyy , Neither the Dunn test nor the Tukey test are appropriate as post-hoc tests for Friedman's test. It's not just a small technical matter. These tests are used in very different situations.

It's quite possible that Friedman test is the wrong test for your situation, as Daniel Wright and Bruce Weaver are wondering.

Unfortunately, many websites present the Friedman test in strange ways, saying things like it's used in the same situations as a two-way anova.

The Friedman test can only be used in cases of unreplicated complete block design. Literally, if you don't have data in this form, you can't get it all into the Friedman test. For an explanation of unreplicated complete block design, and how this differs from a replicated design used in, say, anova, see my response in May 2021 here : https://www.researchgate.net/post/Is_there_a_non-parametric_equivalent_of_a_two_way_ANOVA .

Oleksiy Voytsekhivskyy

adding what was raised by Jochen; Ronan has a point that is so many groups ?.

May I ask if his data is robust?

In particular when working with large amounts of data, but not variables, a couple of problems always appear, First false OUTLIERS (badly typed or badly copied or with error data), Second true Outliers (due to independent variables not considered).

if it detects, you can decide if he uses it or not, but you already know of the existence of data that can alter the mean in some groups.

I sincerely advise you to silver that possibility, in so much data it is very possible that it happens and that is why expected behaviors do not appear.

finally apply the appropriate test for your hypothesis, which can be "parametric" or "non-parametric"

If you have so many groups and it is parametric, Tukey should be a good option for so many groups, Duncan is useful if you want to be quite discriminating, Bonferroni is usually quite conservative and is used mostly when there are not many comparisons to be made and also, the groups are homogeneous in variances. A very good approximation for its calculation consists of multiplying the original value of p by the number of possible comparisons to be made.

in case of being "non-parametric", it would study several options, Moses, Wald-Wolfwitz, or others.

Thanks all for the answers!

Hello Oleksiy Voytsekhivskyy Please note the Friedman test statistics below

S = [ (12n/k(k+1)) * Sumik Ranki ] - 3n3 (k+1) ~ Chi-Square(k-1)

Now pay attention to what happens to the test statistic when the number of k increases. Clearly you will encounter small and significant P value. Without wanting to.

So I think forget the Friedman test or something like that and focus on reducing the number of Ks.

It doesn't matter much in any case, since theory world is very different from real world. Use anyone that is most convenient to you and be done.

I go with jose Risomar sousa, better suggestion and would add use parametric test.