There is some debate as to whether the ANOVA is hyper sensitive to non-normality, but there are a variety of other problems, only some of which are fixed by using nonparametric approaches. Rice and Gaines (1989) is only one of many papers on this problem. There is a popular paper that says the ANOVA is robust, but the authors allowed up to 7.5% to be called significant at the '5% level of significance.'

I believe that a Kruskal-Wallis Test would be better. But such a small sample size might not get much power what ever you use. You could use Wicoxon tests for each comparison with either a Bonferroni or a Sidak correction (they come out the same with such small sample sizes). If you designating one treatment as a control that would reduce the problem of multiple comparisons.

I'll partially disagree with some points by Lewis Maccarter .

1) It's not a great idea to conduct pairwise two-sample tests. One reason is because they ignore the variability of the rest of the data. In general, it is better to use a (post-hoc) test that takes into account all the data together. In the case of the Wilcoxon-Mann-Whitney test, pairwise tests have the weird property that they can report e.g. that A > B; B > C; and C >A. In this case, the Dunn (1964) test is a better option that doesn't exhibit this weird behaviour. But, then, I don't think there's any reason to skip the Kruskal-Wallis test in this case.

2) I'm not sure I would avoid using traditional anova unless there is some clear reason it won't be appropriate. Also note that traditional anova and Kruskal-Wallis test very different hypotheses. It's probably best to start with what hypothesis is actually of interest. ... For a one-way design, there are lots of different tests that could be used, including those by permutation that won't have problems with the assumptions of traditional anova. But also there are, e.g., tests for medians, or approaches that are robust to heteroscedasticity.

Why do you say that you have a block design? Is this true? Or do you have a simple one-way design?

Better to go with one way ANOVA however if the data within a sample is highly variable then one can think of non parametric,

Any way go with One way ANOVA

Hi Salvatore, I respectfully agree to partially disagree. If Yagmur could used pairing in paired tests they would remove (ignore) the side issues that are not part of what is being tested. Paired t tests can be seen as the single pair version of the complete block ANOVA. However, I applaud your alertness that resulted in your noting the fact that the original question was confusingly garbled in that it started with a mention of 'block-design' and then went on to propose a 'one-way ANOVA'. If the data in the question was not somehow blocked then simple t tests of simpler non-global designs (for instance several two treatment tests) would have been more accurate than an ANOVA of three treatments as to significance (and any attempt at power).

I may be long dead before it happens but I suspect that the ANOVA is eventually going to go down in flames. ANOVAs are mathematically very elegant and tempting. However, I think that some day applied statisticians, myself included, will look back at our use of the ANOVA in shame.

Hi, Lewis Maccarter . You have me intrigued. What is it that you think is wrong about using anova? What is it you think will replace it as a "go to" analysis we teach students? ... For me, I always wonder if we will move to 1) a model where we start with a mixed effects generalized linear model. I don't think this will be too difficult to even relatively inexperienced students. 2) a general nonparametric model. Maybe something like Aligned Ranks Transformation Anova or "general factorial design".

Thank you @Salvatore S. Mangiafico for your interest. I will put together a more substantial answer specifically about the difficulties of the ANOVA and post it here this afternoon or tomorrow.

As to what to teach, or as to what to use instead, I am concerned that although using nonparametric tests does not lose as much information, there tend to be serious problems at larger sizes. They and other possible alternatives need more study. So, I think that, for now, we should teach the ANOVA, but with caveats that it is not the end-all-be-all that is commonly thought. I have included it in my own text (with quite a few succeeding caveats) so that students using it can go on with succeeding courses and can evaluate already existing papers. The alternatives you suggest are intriguing. Maybe with robots to help us we could even increase our replication rate so much that we could use some thing like a random forest, or a XGBoost to evaluate our results.

There is also the question of why use global tests in general. If we think about it most questions of real importance are binary. For instance, "Should we give a third COVID vaccination?" needs a yes/no answer. Inferential tests involve an unavoidable chance of being wrong, why complicate the situation? In subjects like agronomy researchers seem to think they must always use multiple treatments, but often a single paired test (equivalent to a two treatment RCBD) would return more for their investment of, often limited, resources. To quote Rosenthal, Rosnow, and Rubin (2000), “Global questions seldom address questions of real interest to researchers, and are typically less powerful than focused procedures.” Although true, that is far from being the whole story with respect to the ANOVA. (to be continued)

Rosenthal, R., Rosnow, R. L., and Rubin, D.B. 2000.Contrasts and effect sizes in behavioral research: A correlational approach. New York: Cambridge University Press.

This follows on to my answer to the kindly query by Salvatore S. Mangiafico above.

The most commonly cited somewhat recent paper in defense of the ANOVA is Blanca et al. (2017) who ran massive Monte Carlo simulations. “In order to examine the effect of non normality on F-test robustness, a one-way design with 3 groups and homogeneity of variance was considered." Like most other studies they focused on Type I error; they did not consider Type II error (the power aspect). However, they used Bradley’s (1978) liberal criterion that a statistical test is considered robust if the empirical Type I error rate is between 0.025 and 0.075 for a nominal alpha level of 0.05 . Thus they went from accepting one false positive in twenty tests to accepting one false positive in thirteen tests. By accepting this, they concluded, "The results showed that in terms of Type I error the F-test was robust in 100% of the cases studied, independently of the manipulated conditions."

A less cited study, also by Blanca et al. (2018), focuses on heterogeneity of variance with respect to Type I error, using massive and comprehensive Monte Carlo simulated data in the ANOVA. They found a necessity to study, and usually manipulate, the variance structure of the data before performing the actual ANOVA. Then in the majority of cases (in some instances the data proved totally unsuitable) the ANOVA, using this pre-manipulated data, was robust (using the Bradley’s 1978 definition of 0.05) with respect to Type I error. They acknowledged that neither their study nor the literature in general considered such matters as power, and they were also aware that some researchers would see this process as unacceptable pre-test data exploration. They concluded:

"Whatever the case, we encourage researchers to analyze the specific characteristics of their design and the data obtained and, if their data do not meet the assumption of variance homogeneity, to choose the best alternative in order to obtain valid results. To this end, the best approach is to perform a simulation study involving the specific conditions of the real data so as to determine whether or not F-test is robust in the situation being considered. We are aware, however, that applied researchers are not usually familiarized with this procedure."

Upon careful reading and even a rereading, neither of these well researched and recounted papers should, as they seem to intend, encourage or justify using the ANOVA. Both papers refer to the simplest possible omnibus ANOVA (only 3 treatments) and both allow too extreme a deviation from even the extravagant alpha = 5% (1/20) maximum; the 2018 paper also proposes a questionable a priori processing procedure that is beyond any but the most math oriented researcher. Indeed I am referring to these two papers as evidence against using the ANOVA for inference. It is likely that, as Blanca et al. extend their Monte Carlo simulations to include power, they will produce additional evidence that the ANOVA should not be used in this way.

I am of the opinion that the conclusions of Blanca et al. (2017) are, perhaps justifiably, grasping at straws on behalf of the ANOVA, a long established procedure, which when confirmed to be inadequate will leave a multitude of those of us who have used it and published its results for decades, with very red faces (MacCarter 2021).

Cited above:

Blanca, María J, Rafael Alarcón, Jaume Arnau, Roser Bono, Rebecca Bendayan 2017.Non-normal data: Is ANOVA still a valid option? Psicothema 2017, Vol. 29, No. 4, 552-557 ( includes 45 citations)

Blanca, María J, Rafael Alarcón, Jaume Arnau, Roser Bono & Rebecca Bendayan 2018. Effect of variance ratio on ANOVA robustness: Might 1.5 be the limit? Behavior Research Methods volume 50, pages937–962 (includes 34 Citations)

Bradley, J. V. 1978. Robustness? British Journal of Mathematical and Statistical Psychology,31, 144-152.

Bradley, J. V. 1978. Robustness? British Journal of Mathematical and Statistical Psychology, 31, 144-152.

MacCarter, L.E. 2021. Research Decisions and Estimation With Confidence and

Power. ISBN-13: 978-1532721076 LCCN 2021913804. Morro Bay, California 568pp.

Papers on the subject of heteroscedasticity relative to the one way ANOVA include:

Rice, William R. and Seven D. Gaines 1989. One-way analysis of variance with unequal variances (data analysis/Behrens-Fisher problem/biostatistics). Proc. Nol. A(ad. S(i. USA 86: 8183-8184.

Go nonparametric

Very interesting discussion. But why not avoiding the whole p-value problem and going hierachical bayesian parameter estimation, as suggested for example by Kruschke (2013). You can do ANOVA-like analyes (or anything else you like) and you will get posterior distributions for all parameters of interest, with wich you can do all sorts of manipulations (e.g. compare or subtract the posteriors of the mu for each group) and different hypothesis testing approaches, as you like (e.g. region of practical equivalence, bayes factor, probability of direction etc.). It also allows to use robust distributions for the response variable (e.g. t-distributions), estimate variances separately for each group, do multilevel modeling or generalized linear models and so on. Why bothering with all the asumptions, if you can model them directly?

Rainer Duesing I think that by suggesting a Bayesian approach, you might be predicting the future. However the present situation is a mess and the average researcher does not really seem to even appreciate the need for statistics. They see it as a black box and a nuisance. Just the other day in seminar (via Zoom) someone quoted his colleagues as saying "why cannot we get on with our research?" Maybe that is why we are seeing articles like this:

Ioannidis, John P. A, 2005 Why Most Published Research Findings Are False. https://doi.org/10.1371/journal.pmed.0020124

A great many of the questions posted here by people with degrees from respected schools would have difficulty even squeezing 80% power out of a t test. Many take the data before even asking what the sample size should be. Some have formed the hypothesis from the data they are testing. I am hoping that we can at least move towards changing the traditional courses one or two steps at a time.

Salvatore S. Mangiafico Thank you for your answer, our experiment design consists of three different blocks in the fMRI task (we presented some photographs to participants during imaging). We want to compare three groups' imaging data for each block. For example, we have beta values of activation (as percentage of BOLD signal change) in a brain region for photograph block 1 (or 2 or 3) (for all participants), and we want to see if there is a difference between the three groups.

Yagmur Karakus , I'm actually more confused now about your design relative to groups, blocks, conditions, and if each participant is in one block, or if each participant participated in all three blocks, and so on. Really the best way to explain what your design is, is to make up some data and just post fictitious data. You have only 31 observations.

That being said, it sounds like you need an analysis more complex than one-way anova or Kruskal-Wallis. If you have one participant participating in more that one block or group or condition, then you probably want to use a mixed effects (hierarchical) model to account for the repeated-measures nature of the data.

It may be that a common mixed-effects or OLS (general linear model) will work for what you're doing. Or if you want to go nonparametric, aligned ranks anova may work. There may also be an appropriate generalized linear model that would be appropriate for your kind of data. I really have no idea what kind of data beta values are.

Yagmur Karakus It's always awkward to obtain data before planning how to analyze it. A Friedman test is often the best choice for small sets of (excuse the lay-term) weird data. However, there is a lot of subtle data in a functional magnetic resonance image (fMRI). Do you have a sizable set of fMRI images available (positive and negative for whatever you seek)? Maybe you could use them for training something like an XGBoost classification and then use your 31 images as the test set.

Maybe check this out:

https://www.youtube.com/watch?v=kqzJLSReg9c

On the other hand, maybe some projects need a stats person on the team (and on the author list) first thing after the literature survey (both before the data acquisition). Good luck sorting this out.