The critical point is that, under some circumstances,* the sample size should not be too small. Larger sample sizes are not a problem.**

----

* when the conditional distribution of the response is heavily skewed

** except for the fact that irrelevantly small mean differences will become statistically significant when the sample sizes are "too large".

To Addisu Tesfaw Zinabu

There are two different cases, as it is mentioned in the question.

In both cases the t-test is acceptable by the standard understanding of the statistics whenever BOTH samples are normal (THUS: independently of the size of the samples)

This algorithm is questionable for small samples if the observable (measured) quantity(ies) is(are) NOT normal. There are many ways possible depending on the kinds of deviations. I would like to suggest performing a QUALITATIVE inspection of the possible distribution of the small sample(s). If there are obvious differences if compared to the normal distribution, and in case you are interested in a very sensitive test - the next step (in this negative case) would be to estimate the distribution by some appropriately chosen way e.g. from two-modal distributions, or, from one-modal but asymmetric distributions. . . .

More details cannot be suggested without writing a book (which i will not, and I do not know a practical guide for this :-D

OR

simple by giving access to some qualitatively the same data of the sample.

Best regards, Joachim Domsta

PS: There are, obviously many other constrains, which canot be handled within brief comments. JoaD

To use a t-test, the sample size DOESN'T NEED to be less than 30. The idea is just that † a) if the sample size is less than 30, you should use a t-test, but b) if the sample size is larger, you are able to use a z-test. This is an antiquated consideration. 1) These sample size cut-offs aren't justified in all cases. 2) Since we have software that does all the hard work for us, usually people will use a t-test even for large sample sizes. 3) The more important distinction between a t-test and a z-test if the if the standard deviation is "known" beforehand.

________________

† Assuming all other assumptions are met.

I'd like to add a remark. It has been a long-lasting debate in statistical community about the validity of t-tests and the associated p values. In recent years, some authors suggested to retire p values. An international journal, Basic and Applied Social Psychology (BASP), has officially banned the null hypothesis significance testing procedure (NHSTP) (p-values, t-values, and F-values) since 2015 (Trafimow and Marks 2015). People are looking for alternatives to t-tests and p values. One of potential alternatives is to conduct a SCI analysis. SCI stands for signal content index. It is a new descriptive statistic discussed in "Signal content index (SCI): a measure of the effectiveness of measurements and an alternative to p-value for comparing two means" https://iopscience.iop.org/article/10.1088/1361-6501/ab46fd/pdf

t-test (distribution) depends on the assumption that the data of the samples are NORMALLY distributed...

If Normality is not acceptable, you must not use it!

I have two comments about Massimo Sivo's statement, "t-test (distribution) depends on the assumption that the data of the samples are NORMALLY distributed".

First, for the unpaired t-test (pooled variance version) to be an exact test, the scores must be sampled randomly from two independent populations* that are normally distributed and have exactly equal variances. In the real world, those conditions are never going to be met, and so in the real world, the t-test is an approximate test, not an exact test.

* Notice that it is the populations, not the samples that are assumed to be normally distributed. And even if both populations were perfectly normally distributed, the samples almost certainly would not be. (They would usually be finite, for one thing.)

Second, all t-tests have a common format (see a cleaner version of the equation in the first attached image):

t = (statistic - parameter|H0)/SEstatistic

The important normality assumption, I believe, is that the statistic in the numerator has a sampling distribution that is approximately normal.

• For the one-sample t-test, the numerator statistic = the sample mean (Xbar), the denominator = SEXbar, and the sampling distribution of Xbar must be approximately normal.
• For the unpaired t-test, the statistic in the numerator = the difference between the two sample means (Xbar1-Xbar2), the denominator = SEXbar1-Xbar2, and the sampling distribution of Xbar1-Xbar2 must be approximately normal.

Normality of the population distribution(s) is a sufficient condition for normality of the sampling distribution of the numerator statistic, but it is not a necessary condition. As n increases, the sampling distribution of the numerator statistic (Xbar, Xbar1-Xbar2, etc.) can still be approximately normal, even if the population distribution(s) of raw scores is/are not normal.

From this point of view, when we use a t-test, we need to think about whether the combination of (raw score) population distribution and sample size that we have will yield a sampling distribution of that numerator statistic that is close enough to normal to make the t-test "useful" (as George Box might have said).

Bruce Weaver

SEE IF you CAN find improvement with t statistics in the attached data!