I think the central question should be: Is it reasonable to assume different variances? -- And if so, is an ANOVA (or t-test) really adequate or is there a better model anyway?

What is your response (e.g. difference/deviation, a concentration, a count, a time until something happens, ...)? Can it be negative? Is the distribution symmetric or skewed? How should it look like? How does it look like?

And if you select the test following the (observed) data, you actually do multiple tests (you did at least two!) to selecte the one with the "best" result. In such a case you would have to correct for multiple testing. It's a bit difficults here because these two tests you made are not independent, but a conservative correction would be to multiply the p-values with 2 (the number of tests). But then, ups, there is no significance anymore... Although many won't believe this, but a p-value refers to the entire process how the conclusion is derived, and it must consider the possibility of selecting among different tests (and transformations). The only way to circumvent this is that test (and transformation) are given through former experience (other, independent data) or by theoretical considerations (like: the response are counts, and the rates in the Poisson process can not assumed equal in the individuals, so I will use a negative-binomial model to fit overdispersed count-data).

In my opinion, the Welch test does not assume different variances. It is constructed in a context where variances may be equal or different. If the variance equality assumption is valid, it becomes a good approximation to Student's T-test (theoretically, the most powerful test in the homoscedastic case). If sample sizes are equal or similar (n1 = n2 +/- 2 or 3), T-test is preferable even in the case of moderate heteroscedasticity. In other cases, Welch's test is preferable.

Jorge Ortiz

Sara: with sample sizes of 5 and 30, I think it is preferable to apply Welch's test and don't worry about the variances difference unless this is a second parameter of interest. With so different samples sizes, the tests for comparing variances are very dependent on distributions and don't respond adequately in not normal case (Conover has studied this problem several years ago)

With one of the conditions having only 5 cases the question is how do you want to use this (limited) amount of information to form a n estimate for that condition's variance (i.e., if using only data from 5 cases the variance estimate won't be very reliable). The Welch and Student t-tests are at far ends of the scale (I guess further than Student's would be just using the bigger group's variance) on whether to use information from other groups. As Jochen Wilhelm states, it is best to think about what is reasonable for your circumstance. It is very important not to decide just on the basis of which is significant (this Article Increasing Transparency Through a Multiverse Analysis

James Byron Nelson Interesting!

I see people have different opinions on the matter. Of course, with a small sample size of 5, there is no ideal situation, unfortunately.

I follow answers

best wishes

If you compare such small groups you should use non-parametric test according to my opinion. If you compare two independent groups then Mann-Whitney is the test of choice. Nota bene T-test is different from t-test!

S. Béatrice Marianne Ewalds-Kvist, could you specify why a non-parametric test would be better, particularily for small samples? Why do you think is it ok to test a different hypothesis just because testing the original hypothesis desired to test is a bit more complicated then simply applying a t-test?

Hi Jochen,

Glad you comment Jochen. In Finland if n

Dear S. Béatrice Marianne Ewalds-Kvist

In my experience non-parametric tests have less power and it much harder to find signficance. The only benefit of a non-parametric test is that it does not assume that the variables are normally distributed and you can use it for other than continous data (for example: ranked data). So perhaps it will be easier to find significance when data is clearly non-normally distributed. But in other cases, I prefer the parametric tests , even when sample size is small (although I remember in our statistics class the professor also mentioned the

Dear Sara,

wait for Jochen's comment about this issue. My experience differs from that you had.

Hmm, it seems now I'm in duty...?!

I hoped to get some argument other than "because others do that". There is a confusing literature about this topic and you categorical "use non-parametrics" seemed as you knew something more.

"It is though somewhat easier to get a significance with non-parametric methods according to my experience" -- this is well known, at least for sample sizes that are not too small (!), when the actual distribution is skewed. This is because the values far away from the mean at one side inflates the variance, and the non-parametric tests do not consider the variance. But the cruicial point here is that these two significances (from the to tests [e.g. t-test vs. MW-test]) refer to different null hypotheses that are tested. To exaggerate this: that's like comparing two risk statements, one being larger than the other, and therefore concluding that the one statement is to be preferred over the other. However, one statement is about the risk to get a cold, and the other is the risk to get insured in a traffic accident. It does not really make sense to compare the numbers to find which risk assessment is "better". The important points left completely out what - in the given context - is relevant: catching a cold or having a traffic accident?

There is also another aspect: (i) selecting a test based on the data is then part of the procedure, but this get unconsidered by the test, so that the p-value loses it's usual interpretation. (ii) if we really had the choice between different tests, we should actually use the one that will make it harder to become significant. We should always try find the most critical and severe test. And among all equally critical tests we may then chose the one with the highest power.

Sara is also right, because for very small samples, the power of non-parametric tests is low. A reasonable assumption that can be used by the test is then helpful. But again, one has to ask what the effect is for which the power is given. Again we are about to compare catching a cold and having traffic accidents.

So before discussing and comparing these different tests against each other, we need to be clear about what we want. These tests are (usually) about different hypotheses, and being able to reject one does not mean that the other is also to be rejected. If the answer is "it must be significant (who cares about the hypothesis being tested)", then we should not call ourselves "scientists", imho.

Wilcoxon test I did not suggest because Wilcoxon is for related samples.

As you can already see from the figure the SD.s are big and the difference is not likely to be significant. The researcher should of course decide beforehand what test to be used but this a normal thing on RG that someone needs a suggestion at this stage of their research. What does imho mean?

imho = in my humble opinion

Sorry that I mentioned Wilcoxon as an example. You are right, and thank you for pointing this out. I changed it to MW-test to not add more confusion.

I agree that people seeking for suggestions. However, I don't think it's constructive to throw in "general advices", because this does not really help lastingly. It may be a "quick help", but to my experience this eventually only piles up confusion and makes statistics to something strange with wired and obscure rules which are sometimes applied for unknown reasons.

I think small sample size results in large variance for the first Group (as it is clear from the graph), which makes the two CIs overlap and as expected in this case a parametric test may not reject the hypothesis of equal means. Excellent explanation from Prof Jochen Wilhelm .

https://www.cscu.cornell.edu/news/statnews/stnews73.pdf

Jochen,

Thank you, imho:)

Inho (FI) means disgust (Abscheu)

Béatrice,

so I am happy not having made a typo there :)

This is what their argue in Sweden and it sounds odd to me!

As an aside: nonparametric tests don't in general have lower power than parametric tests. The asymptotic efficiency of most common nonparametric tests is typically around 95% if the assumptions of the parametric test are met. So there is typically a slight loss of power (with some exceptions like the sign test with lower efficiency). However, if the assumptions of the parametric test aren't moderately or seriously violated then the parametric test may have much greater statistical power.

That doesn't mean one should always use nonparametric tests (though that argument can be made), but I think they can be sensible options particular for small samples. In many cases I'd consider a generalised linear model or bootstrapping before switching to a nonparametric test (such as the common rank transformation ones). This is because a suitable parametric model typically has clearer connection between the original data and the model used for inference.

An aside-aside: non-parametric tests test different hypotheses. Only for n -> Inf, they (often) are often approximate tests on the same hypothesis - and just then violations of assumptions the parametric tests make are becoming irrelevant either. My concern is that the "power" in these different tests also refers to different hypotheses (at least for "small" n), and I don't see how these can be compared. A power of 90% in test A can have an entirely different practical meaning than the same power in a test B, particularily for "small" n.

I fully agree to the recommendation to use appropriate models including an appropriate error structure with parameters that do have a meaningful and useful practical interpretation.