Well, question is I think difficult since with the K-S test you can compare distributions that do not even have a mean... Questions asked by the two tests are really different, so I do not think a simple relation exists between the two. Especially if, by « means equality test », you think of the usual Student's T test.

If you think of special context, simulations may be the best way to explore the behaviour of the two tests.

By the way, I do not see why the two hypothesis could not be accepted at the same time, since the second one involves the first one so there are necessarily cases in which both will be accepted!

Often populations need to be compared that are (approximately) normally distributed. In this case it is advantageous to use a method that takes the normal distribution into consideration as

- it gives higher power

- allows you to express the difference (and its uncertainty) in an easy-to-understand from.

Dear Emmanuel, thank You for the answer! Yes, I use Student's T test as a test of means equality. For example i considered two samples, one of which was generated by uniform law and the other by normal, both of them with equal means. T-test gives H0 hypothesis as far as K-S test gives an alternative - H1, I attach the file with this example. This is an example, but I faced such effect during my work with experimental data.

On he other hand the mean, if it is exist and if the law of the random variable is continious, is the Integral xF'(x) dx and the K-S test uses an estimation of F(x) given by the set. So speaking in general I think it can be the dependence between statistics that these two tests use.

Dear Gabor, thank You for reply! During the data analysis I rather oftenly faced data sets that does not look like normally distributed. When i can say for sure that data is normally distributed, at that moment I feel myself as a happy person!

> Petr: well, if one of your sample is strongly not normally distributed, and the two samples do not have the same variance (unless you specifically choosed the uniform law to match the normal variance), it is really not a surprise that the T-test gives different results than the K-S tests, since it is simply meaningless (application conditions are not met...). Note also that you have in your example the same mean and different distributions, so your results with T-test (H0 : same mean) and K-S test (H1 : different distribution function) are exactly as expected (you are lucky with this T-test, the sample size certainly plays in it) and I must confess I do not see in that case where is the point...

As for the second point, since both tests will use the same sample, there will be a relation between their values; but to express it and decide in which conditions they will differ does not have a simple answer I think... And adding that K-S test is asymptotic, it is even more complex. If you want a try, just try with n = 2 or n=3 sample...

Emmanuel, of course the example that i mentioned is only test. Practically I usually have a deal with two or more sets about which i know nothing before starting the tests. And sometimes t-test and K-S test give me not equal answer on the question "does both of the sets have the same origin?". Of course much more often K-S and T both gives H0 or H1 hypothesis at the same time. That's because I become interested if there is known dependence between both of the statistics.

What do you mean saying about the sample with n=2 and 3 - is it number of different sets to test? For the number of observations that one set contains it is two small, I guess...

In brief: K-S test and T-test do not answer the same question, unless you know both distribution functions are Gaussian (« normal ») with same variance, so definitly you have no reason to expect the same conclusion for both.

Since K-S test is not developed to focus on the origin problem, you cannot use it reliably for such questions. If this is your concern, you may use instead Wilcoxon-Mann-Whitney test (with caution) or bootstrap methods.

For n=2 and so one, I meant sample size to try to find a analytic link between the criteria of the two tests. If for so small sample sizes it is difficult, imagine with real sample sizes...

Emmanuel, of course these two tests answers on different questions. And it is not surprising that K-S test can give H1 hypothesis while t-test gives H0 as it was in example attached above. But I have met more contradictory result - when t-test gives negative result (H1) and K-S test gives positive (H0). It is rather strange situation when I have to accept hypothesis that two random variables have different means but the same law (including all parameters of course). What hypothesis should i accept in this case?

I'll attach such example this evening to show what i mean.

I am afraid that with sample size n=2 K-S will not work because it is impossible to make intervals and consider the cumulate function in reasonable way. T-test will work in this case only if both of variables that generated samples have normal distribution. That's because it is only with large n when the sample mean of not normal variable has distribution that is close normal (that is according Central Limit Theorem).

By the way, one of K-S authors N.V. Smirnov was sorry in his book (Большев Л.В. Смирнов Н.В Таблицы математической статистики, 1983 (3rd edition), p.80, I do not know if it was translated to English, so excuse me for book title in cyrillic) that some other pepople use K-S test with unreasonable small n. In mentioned book, page 80, he recommend to use n not less than 20.

Thank You for advices about Wilcoxon-Mann-Whitney test!

Nada, using t-test you can have samples that are not originally normal. As i know, T-test works with any sample distribution law when sample size is big enough. That's because of assymptotic normality of sampe mean statistic when n is big.

Thank You for Your advice with Shapiro-Wilk test!

Thank You, Nada! I'd better say not about test power but about probability of Type I Error or Type II Error in application to specific situation.

So here is example (2 already sorted samples) where with confidence level about alpha=0.07 we have to reject H0 hypothesis "populations have same means" according to t-test and at the same time accept H0 hypothesis "populations have same distributions" by K-S test. Number of intervals in K-S test was assumed as 15.

There are several things in Petr and Nada posts... Here are some comments.

1) Petr seems to use KS test to compare two empiric distributions, whereas Nada seems to use KS test to check normality. This is two different things.

The KS test to compare two empiric distributions is asymptotic, hence the condition on n. Indeed, for small samples, one must compute the exact distribution law of the criterium; it is known in some cases, I am not sure for the case of two empiric distributions.

The KS test for normality must be used with caution, since it expects the mean and variance of the normal ditstribution to be exactly known, and not estimated, which is usually not the case. In such case, corrections must be made, such as in the Lillifors test. Besideds, the Shapiro-Wilks test is known to be more powerful to detect deviation from normality, even for n >= 3 (For n=2, I don't think trying to asses any distribution law is realistic anyway)

2) The "n > 30" => I can use T-test whatever the distributions are is really an over-simplification approach. It depends on the real distribution. Besides, uses of the TCL ensures that alpha is correct but leads to an important loss of power (because it does not hold under H1...) so it is not really useful, compared to non-parametric tests for instance, if (strong) non-normality is suspected. Some of Petr results may come from this loss of power, and indeed illustrate them.

3) The equality of variance is another condition, that should be tested if no correction for that is made (as in the classical Students's T test). Loss of robustness is more sensitive when sample size are very different, as you mention, but even when they are comparable, this should be checked...

4) Petr: I am not sure of what you exactly want to proove, but I would say that selecting H1 is the positive result and failing to select it is a negative result, since a test is made to reject H0 (hence "prooving" H1) and not accept it [you may read the comments of Jeffrey Welge,Jochen Wilhem and I on another post for more discussions about this]. If you use T-tests or K-S tests, you want to proove a difference; selecting H1 is then a succes when distributions are indeed different by mean.

In that sense, KS tests rejecting H0 and T-test not is not contradictory: T-test is not powerful enough to detect the change (see above for some reasons), whereas KS is. Always remember: non-rejecting H0 is NOT accepting H0 !

The other way round applys also: T-test focuses on mean differences, hence can be more powerful to detect changes that are "only" on this parameter than the K-S test that must handle with any change in the distribution function. In other words: tools for specific questions are more powerful; to answer more general questions, you have a price to pay and are less powerful. That is, in fact, why there are several tests and not a single universal tool...

Hope this helps,

Emmanuel, thank You for detailed and constructive answer! It is clear that all of these tests answer the question "is the sample data contraries our H0 hypothesis". But I'd want to clarify to myself one question.

Let we have two samples - x1 and x2, both of the equal size n, and n is rather large. I used the next statistic of t-test:

t=(sample_mean(x1)-sample_mean(x2) / sqrt(variance(x1)/n+variance(x2)/n)

If x1 cosist of independent observations from population(x1), the random variable "sample_mean(x1)" is distributed normally if n large enough. It has mean population_mean(x1) and variance population_variance(x1)/n.

Near the same we can tell about random variable "sample_mean(x2)".

So if n is large enough (for any population distribution) and if populations for x1 and x2 have the same mean, the numerator is random variable with zero mean independently of the variances.

Further we can say that if observations that make samples were received independently, the variance of fraction numerator is (population_variance(x1)+population_variance(x2))/n. So if n large enough, numerator distributed normally with zero mean and some variance, while all fraction is distributed normally with variance=1.

By the way, it works also for samples of size n=2, from normally distributed populations with equal means.

So if examining my x1 and x2 i meet large meaning of numerator i should suppose one of two things:

1) populations for x1 and x2 can have the same mean if populations variances is VERY large

2) means for populations are not the same.

As s^2 is the consistent estimate for population variance, the probability of the fact that s^2 differs from population variance greately for large n is very small.

So this simple test show me (in example2 above) that populations has different means with probapility that is close to 1. Then K-S test uses max(sample_cumulative(x1)-sample_cumulative(x2)). And in my example i see that cumulative functions are slightly shifted one to each other. Because the shift is not large but constant, max(sample_cumulative(x1)-sample_cumulative(x2)) is not large, but t-test shows the means difference.

So what the final decision should I do? Populations means differs, that's for sure. But what about K-S result? Two samples have the same type of distribution but their means differs? Of maybe distributions differs at all and K-S result is because insufficient sample size or improper intervals making?

Well, assuming n is large enough, as you assume for previous T-test, the second option may be ruled out... However, the first one seems a good choice: there is a very slight _horizontal_ shift of the CDF function when mean changes ("same type of distribution"), but KS-test is much more sensitive to vertical differences in the CDF so is not powerful enough to detect this.

To check this, you may try CDF-comparisons tests that are based on other measures of discrepancy, like the Cramer-von Mises tests that, IIRC, measures the area between the two CDF, so should see also a small horizontal shift.

Not however that since T-test is, if I remember correctly, the most powerful (when normal distribution assumption holds at least), you can always have the T-test significant and any other test unconclusive...

Emmanuel, thank You very much for Your help and kind advices! I hope all of these will help!

can I pose a questione related to two set of groundwater sample? so I have two set of chemical concentratrion distribution related to 185 groundwater points of control, exist a statistical procedure or analisys to compare these two set to find a statisical correlation? Thanks and sorry for my interference [ah, I'm not an expert statiscian]

> Antonio : 1) you should open a new question and 2) the answer is certainly yes, but without more details on your data and exact question, no answer can be found.