A better option would be to carry out Kruskal Wallis test, which is a non-parametric alternative to ANOVA. I have two publications where I have used this test.

Article Online learning experiences of social work students in India

Article Wellbeing of Palliative Care Workers During Covid-19 Pandemi...

s. Rama Gokula Krishnan can you please explain why? Non-parametric tests do not answer the same questions/have not the same hypotheses. Why should one switch to them, if there are good robust alternatives, if variances are unequal or other issues. Please demonstrate why your approach is more suitable?

• It's generally a bad idea to decide which test to use based on another test (Levene's). You'll find a lot of discussion on this and other forums. One brief piece that discusses this topic with testing for normality is here: https://towardsdatascience.com/stop-testing-for-normality-dba96bb73f90 . If you have a question about homoscedasticity, just plot the residuals from the analysis (vs. predicted values) and have a look. (or in the case of a one-way design, just plot the data by group.)
• There's no harm in using Welch's anova for all these. This article discusses this in terms of Welch's t-test: http://daniellakens.blogspot.com/2015/01/always-use-welchs-t-test-instead-of.html . It's also fine to use Welch's for some and non-Welch for others. I think this comes down practically. Will your audience question why you didn't use standard anova when variances were close to homogeneous ? Or, is it just easier to say you used Welch's and not have to explain further ?
• I don't know specifically if there is a loss of power using Games-Howell vs. Tukey. I imagine if there is, it isn't enough to worry about.
• Personally, for the ease of explaining, and the ease of doing repeated analyses, I would opt to use Welch's and Games-Howell for all.

Rainer Duesing I just suggested a non-parametric alternative because the Kruskal Wallis test is the least affected by unequal variances when compared to F test/ANOVA

Article ANOVA, Kruskal-Wallis, normal scores and unequal variance

s. Rama Gokula Krishnan but non-parametric tests do not answer the same questions, have a huge information loss and there are appropriate parametric alternatives, if specific assumptions are violated, e.g. Welch for heterogeneity, trimmed approaches for heavy tails and outliers, bootstrapping for non-normality etc.

The reference you give in your papers, Heck (2012) does not take that into account. Besides that, the different power analyses do not really convince me. 1. the differences are not so huge (but this is subjective), but in favor of the Kruskal-Wallis test and 2. she uses a log-normal distribution, for example... if I know from my data generating process (the population) that the noise distribution is log-normal, I would use an appropriate method to account for it and not blindly apply a t-test or ANOVA. So in my opinion it is a comparison of the 2 bad choices to analyze the data in the first place....

In your first paper, you explicitly state that the data is ordinal, so it does not have anything to do with the distributional form, but the measurement scale and is not an appropriate exampl imo. Although ordinal regression or similar approaches could have been used.

Yeah, simply put, if heteroscedasticity is the problem, there are various ways to approach this that maintain the test of means, as in anova, and without resorting to non-parametric approaches.

• I think what OP suggests for a one-way design makes sense: Welch's anova with Games-Howell post hoc.
• A more general approach is to fit the correct model, and then use emmeans as post-hoc. This might mean a generalized linear model, or generalized least squares with weights. No problem with modern software.

I've been wanting to write something up on this.

• Using Kruskal-Wallis under conditions of heteroscedasticity is not as simple of a discussion as it might seem. Some authors state that homoscedasticity is an assumption of using the test. As far as I understand: This isn't necessary, but it simplifies the assumptions. Because you can get inflated type-I errors if the population groups are stochastically equal, but have different variances. Conover has a nice solution of writing the assumption as, either one group tends to have higher values than another or the they have the same distribution. They have to be exchangeable under the null.

Lauri Erkkilä, please report the following for each of your 4 groups:

• n
• Mean
• SD

Thanks!

Rainer Duesing I don't understand as to why people are very biased towards non-parametric tests. These tests are simply more powerful than their parametric peers when certain assumptions are violated. Now, if one wants to ignore those assumptions at their own risk of incorrectly accepting the research hypothesis, then nothing can be said.

Universally, non-parametric tests are considered to be a suitable alternative to parametric tests under certain circumstances. I suggested the Kruskal Wallis test because OP's goal was to simply compare two groups and one of the main assumptions of ANOVA test wasn't met.

In my paper that you mentioned, I have opted for the Kruskal Wallis test not only because the variables were ordinal in nature but because they were also not normally distributed.

At the end of the day, it's the decision of the researcher to choose which test would be most appropriate to test their hypothesis.

s. Rama Gokula Krishnan, you wrote:

I suggested the Kruskal Wallis test because OP's goal was to simply compare two groups and one of the main assumptions of ANOVA/Welch test wasn't met.

Which assumption do you believe was violated? And why do you believe it was violated? I ask, because IMO, Lauri Erkkilä has not said enough about her data for us to judge how well the assumptions of ANOVA are met or not met. That's why I asked her to report n, Mean, and SD for each of the 4 (not 2) groups. It would also be helpful to know what the dependent variable is.

Bruce Weaver Since OP highlighted the Welch test which is usually used when the variances are not equal, I assumed that her data has this problem. Hence, the suggestion to use Kruskal-Wallis test instead of ANOVA.

As far as your question with regard to the dependent variable is concerned, I was referring to my study, not hers.

Your suggestion to report n, Mean, and SD is also good.

His* :D

I have hundreds of parameters to compare between the groups, and in some of them Levene is non-significant and Welch significant. The n's are around 90-70-25-10.

Hello s. Rama Gokula Krishnan. Thanks for your response. You may be interested in the simulation study by Zimmerman (2003). It shows that the Wilcoxon-Mann-Whitney test is not robust to variance heterogeneity. Granted, that is the 2-sample case, and I am not aware of any similar simulation studies on the Kruskal-Wallis test. But I would be leery of using KW to deal with variance heterogeneity. Why would you not use Welch's F or the Brown-Forsythe F instead? Neither of those requires homogeneity of variance. (Both are available via the ONEWAY command in SPSS, for example.)

PS- Here is a short conference presentation I gave several years ago discussing Zimmerman's (2003) simulation study.

Presentation Silly or Pointless Things People Do When Analyzing Data: 2. ...

s. Rama Gokula Krishnan as already stated above, my two main issues are that rank based methods do not answer the same questions as metric tests, like ANOVA or t.test. They compare means/medians only under very specific situations, please address this issue.

Second, you did not address several proposals for different approaches and methods to tackle issues with heterogeneity with paramteric tests. Your only argument was so far that under specific violations (non-normality, c.f. Heck (2012); heterogeneity, c.f. Lachenbruch & Clements(1991)) K-W test has higher power than a normal, uncorrected ANOVA. But what about the Welch correction, OP was asking about?

So, I did two small and very simple simulations (I did not systematically vary all parameters, but it is sufficient to get a grasp and run it with different manual changes), which you can find in the attached R script.

The first simulation addresses the results from Heck (2012), where I sampled three groups (Cohen's d = 0.5) from a lognormal distribution. I added the Welch Anova, trimmed Anova and bootstrapping as methods. I can confirm the results from the original study that KW outperforms the others in terms of power, BUT (a huge but), as stated in a former post, all of the methods are not appropriate in my opinion (the metric because normality is violated, the rank based because it answers different questions, I am typically not interested in). Therefore, if I do know a priori that my data generating process produces lognormal data, I adjust my analysis. Here, the simplest way was to take the log of the dependent variable (one could additionally use a Bayesian approach to model a shifted lognormal likelihood, but this would have gone too far...). If I conduct the same analyses (now on the log(lognormal)--> normal data), the results change dramatically. Now, ANOVA, Welch Anova and bootstrapping show a higher power as compared to K-W on the lognormal and normal data (which is the same).

The second simulation addresses heterogeneity on normally distributed data, for which parametric tests with corrections should work fine. Here we can see that (as expected) Welch Anova shows the highest power, especially in the worst case, when the smaller sample has the larger standard deviation. It is not a universal pattern, depending how equal the sample sizes are, but the other test also perform better than K-W test, especially trimmed means.

Therefore, we may conclude that an automatic shift to non-parametric tests is not a good solution, should not automatically be done and cannot be justified with a power argument. Look at your data and the data generating process and find an appropriate test. Sadly, there is no cook book recipe which can always be applied.

Rainer Duesing , nice work.

Bruce Weaver , I understand the Zimmerman in context of your presentation, and in context of this discussion. But, in fairness, it's starting with the premise that "no difference" means "equal medians", which isn't really fair to the WMW test... I have some comments above (29 Nov.) about the assumption (or not) of homoscedasticity for WMW or K-W.... I'm pretty sure I did some simulations on this at some point, but I'm not definitely not organized enough to find them quickly....

Good point about medians, Sal Mangiafico. I must confess that I have not read Zimmerman (2003) recently, nor have I reviewed my own presentation recently. So I might make a few updates if I do. ;-)

Bruce Weaver , Zimmerman makes sense when someone is trying to use WMW as a "substitute" for a t-test. But that's the underlying problem. It's a different test.... There's still an interesting question about heteroscedasticity in WMW. I have a few comments above (29 November, last bullet point).

Sal Mangiafico, I've just taken a quick look at my slides, and you are right: Zimmerman (2003) simulated situations where the population medians were always the same. But in my own defense, these two bullet points are from my slide 10. ;-)

• H0 for the Wilcoxon-Mann-Whitney test often described as saying the two population mediansare equal
• But that is true only if it is also assumed that the two population distributions are the same [including homogeneity of variance!] apart from a shift in location

So yes, Zimmerman's results apply to the WMW test when it is used as a test of location under the pure shift model, as Fagerland & Sandvik (2009) called it.

Article The Wilcoxon-Mann-Whitney test under scrutiny