Hello Afton,

Try bootstrap/resampling estimation instead of the usual F-test for the anova; doing so allows you to avoid concerns about specific distributional properties and homogeneity. Do note that, if you use the Kruskal-Wallis, homogeneity is presumed.

SPSS has bootstrap as an option. R offers several libraries which include it (c.f., https://cran.r-project.org/web/packages/lmboot/lmboot.pdf). SAS can do bootstrap via do loops (https://www.lexjansen.com/phuse/2005/pk/pk02.pdf). Stata may also be set up for this (http://www.philender.com/courses/linearmodels/cnotes4/ranova.html)

Good luck with your work.

Your question seems to say that you don't want to estimate a parameter, like the trimmed mean, geo. mean, etc. Are you set on using non-parametric procedures as opposed to say using transformations, bootstrapping, etc.? You might also tell readers more about your data. Further, the distribution of your variables is not an assumption, just of the residuals.

What is your sample size?

How big is your pvalue?

If the data IS indeed normal small pvalues are possible

Using test to assess non normality is known to be problematic and known to increase false positives in further analysis

Also the fact that you are not looking for an estimate makes me question your research question

If you want to use Wilcoxon or kruskal wallis bc "your data is not normal" then i have even more questions, because the outcome of those tests is often misunderstood

Daniel Wright I am not set on the method, but I have been exploring my data for a while now. I thought to run a one-way ANOVA and was going to use a regerssion to account for covariates if there was a different between the means of the groups. I ran my assumptions tests (ran on the residuals) for outliers (interpret_outliers), normality (shapiro.test), and homogeneity (leveneTest). My data violates normality and homogeneity.

Initially (many months ago), I planned to run a hierarchical regression (continuous DV, continuous and categorical IVs and a grouping var). When doing so, I ran the assumptions on my residuals of each model, I violated both normality (shapiro.test) and homoscadasticity (Pagan-Breush test). I used a log-transformation to account for normality and a least weighted squares correction on the models. When I reassessed the assumptions and compared the model with the transformations, the assumptions were still violated and the anova() between the models indicated the 8th and 9th model had degrees of freedom below 0. I assumed this meant the model assessment I had planned was too complex with my non-normal data. Thus, I simplified to the ANOVA and planned to use the regression or ancova only if there was difference between groups. While this did not solve the issues of violations, it did simplify the test. That is where I am at currently in which test to run, given that the transformations I tried on the regression gave uninterpretable results.

David Morse

Many thanks for your answer. Would bootstrapping being a similar transformation to log-transformation or least weighted squares? Or, have I misunderstood resampling methods?

I have tried the ANOVA.boot() method from the lmboot package in R. I ran the function, p-values, and the summary. Although, I am not really sure what it would suggest. The p-value was 0 and running the data showed a list of the 1000 resampled values.

How big is your sample? Also, you say it did meet these assumptions. How are you assessing these? Are you using p-values? Overall it sounds bootstrapping would be the best approach. Here (Article Using Bootstrap Estimation and the Plug-in Principle for Cli...

You still won't share how small your pvalues were. With a big enough sample size those tests will always tell you data is not normally distributed!

Did you plot the distributions?

Don't rely on tests so blindly please

Hello again Afton,

Bootstrapping is not an analog to a transformation (such as log). Bootstrapping can be applied to ordinary least squares methods, weighted least squares methods, or a host of other methods, including non-parametric. The scheme, by itself, isn't an analog for any of them.

I think you'd benefit from seeking help locally with applying and interpreting the results of a bootstrap solution (or whatever solution you end up using). I'm sure your university has a number of folks who would be more than qualified to help you sort out the best path for your study.

Good luck with completing your project.

Stefano Nembrini

Apologies, I had not seen your response when I was commenting back. I have no issues sharing the test outcomes or qqplots. I simply did not see your original comment.

• N = 620
• p for shapiro.test = p-value < 2.2e-16
• qqplot see attachment

Daniel Wright

Many thanks for the resource!

To answer your questions,

N = 620

p = < 2.2e-16 with shapiro test

Afton Nelson, I may be wrong, but your sample data appears to be heavily skewed left, and set within a bounded domain in which many measurements are situated near the 100 maximum. The shape, to me, seems to resemble a negative exponential or beta probability density function, though this is probably only superficial.

An example of this kind of population would be the test scores on an exam in which most students did very well. If you take an average of those scores, you would say that the class was an unqualified success, since the central limit theorem indicates that the scores were distributed comfortably around a B+/A-. But what about those poor kids that got a 15? Nobody got an A+++.

If the spread of scores is saturated near the maximum, the measurement instrument itself may be the cause of the truncated distribution. Transforming the data would just introduce bias.

The crucial question is not what test to use or even normality, it's what is the data type in your experiment? It appears to be ordinal with finite bounds. The most important question, of course, is what is the experimental hypothesis?

And what were the prior informed real-world-value based metrics by which the collected data would be evaluated, given differences of some sort would naturally arise? The data does not really work for an exploratory analysis here, in my opinion.

Again, I may be wrong on all counts, and others may provide better insight. I'm not a social scientist.

With n=620, the concerns of non-normality are less than for smaller samples, though they still exist. The options seem transform, use a different error distribution (and Miky Timothy says), or use the observed empirical distribution (bootstrapping). I like Miky's response as it stressing the need to consider how the data arise. That should inform the choice.

I meant a plot of the data not of the qqplot

Instead of transforming and then having to deal with interpreting the results after the transformation i would probably give quantile regression a try

Miky Timothy

Many thanks for your thoughtful response. The variable is, as you mentioned, any value between 0 and 100. I was treating this as continuous, given that there were no set boundaries (not grouped by percentiles). The variable itself is the percentage of confidence one has. The idea is to assess confidence across cultures, in which there is little to no previous literature concerning the topic. The idea is to simply see if reported confidence (given in terms of a percentage, rather than in words such as 'very confident'/'confident'/'not confident'/etc.) varies between groups. This was the purpose of the 'simple' comparison in using the means of reported confidence. Many reported high confidence explaining the saturation toward the maximum, so it is rather their reported 'scores' than some produced by a test/task. As I mentioned, I only opted for this 'simple' assessment to get a baseline of whether this reported measure varies across groups.

Beware of nonparametric tests as they often require more assumptions that parametric ones, also most nonparametric test will give you a statistic which is not straightforward to interpret

If your variable is bounded like that why not use a beta regression?

Yes: beta regression or quasi-binomial regression (it does not seem that you need the full likelihood).

Alternatively, you might use a logit-transformation of your data (works if there are no values exactly on the boundaries); the logits should be approximately normal distributed.

In any case, make sure to rescale the variable to be in (0, 1) (that is: divide the values by 100).

If you do have values at the boundaries (0 or 100), you can add a starting value, so something like the Yates' correction from ChiSq would be add .5 and then divide the sum by 101. Different values could be used, eg., 3 and divide by 106. If this were something like grades this wouldn't make them normal distributed with the typical US grading schemes (i.e., the A > 90, B > 80, and a handful of 0s for those not turning something in).

ps. Actually working on a paper with confidence right now. People took a quiz, made a confidence rating for each item, as well as an overall guess of total number right at the end. Our conditions had some people justify their answers (confidence dropped, and so did diagnosticity).

Jochen Wilhelm and Stefano Nembrini Many thanks for this excellent suggestion, which I hadn't considered. Would you happen to know of any particularly helpful resources for beta regression? I have not worked with it before, but I did read through the Beta Regressions with R article as a start.

I also noticed Daniel Wright 's suggestion to add a starting value like with Yate's correction. Instead, would it be possible to adjust the values slightly from 0 and 1 (0.001 and 0.999) based on this formula: x′=x(N−1)+sN " where N is the sample size and s is a constant between 0 and 1. From a Bayesian standpoint, s acts as if we are taking a prior into account. A reasonable choice for s would be .5" (Verkuilen, 2005)? I found this suggestion on Stack Exchange. Alternatively, why would one not opt for a zero-one beta inflated model? Is it because it is more difficult to interpret?

If you don't need the likelihood (for AIC etc), you could simply use base R that provides quasi-binomial GLMs (see attached PDF)

Jochen Wilhelm Wow, thank you greatly!! This is such a helpful template. I got down to the logit/inverse logit section... using the example, could one say that the mean percentage of group1 is 7.3% whereas the mean percentage of group2 is 19.8%? And concerning the log odds ratio, could one also then say that group 2 has 1.14 times greater mean probability (or percentage) than group 1?

Yes, 0.073 = 7.3% amd 0.198 = 19.8%.

To compare these proportions via log odds, you must consider the odds, not the probability. This is the key feature in beta-regression (or quasi-binomial models). You are comparing odds, not probabilities.

OR1 is the (observed) odds of 0.071: 0.073 / (1-0.073) = 0.079 or about 2:25

OR2 is the (observed) odds of 0.198: 0.198 / (1-0.198) = 0.247 or about 1:4

The odds for sample 2 are higher than the odds for sample 1; it is about 3.14-times as large as OR1. This is tha odds-ratio (odds of sample 2 versus odds of sample 1). The slope-coefficient of the model is the logarithm of this odds ratio: log(3.14) = 1.14.

The fact that this value is positive (and not negative) tells you that the oppd, and therefor the probability, in sample 2 is larger than in sample 1. But is does NOT equal the ratio of probabilities. The ratio of probabilities is

0.198 / 0.073 = 2.71. This is a very different number. The probability in sample 2 is 2.71-times as large as the probability in sample 1 (not 1.14-times greater).

It's not possible to get the ratio of the proportion by the ratio of the odds, because there is no 1:1 mapping.

It makes more sense to say that the odds in A are x-times the odds in B.

Afton Nelson

you mention that your result is

N = 620

p = < 2.2e-16 with shapiro test

please be mindful that Pvalues should always be expressed as a number rather than smaller than a value. There should be a way in your software to extract that information