The results are correct. What may be wrong is your interpretation of what p-values are or what they would mean.

This is exactly what tests do: under the null-hypothesis they produce a random number, uniformly distributed in the interval [0,1] (= the p-value). The distribution of the p-values becomes right-skewed when the null hypothesis is false. The bigger the difference between the data and the null hypothesis, the more skewed is the distribution (this can be expressed as the "power" of the test).

This is also the reason why a single p-value can never be seen as evidence against the null hypothesis (or for the alternative). A samll p-value is a small p-value. What would be interesting is if this small p-value comes from a uniform distribution (the case where the null hypothesis is true) or not. There is nothing what can tell us this. Only the likelihood that the small p-value comes from a uniform distribution is lower than the likelihood that it comed from a right-skewed distribution. Therefore, in the long run, when repeatedly the null hypotheses are true, we would expect only fwe small p-values. Thus, p-values may be used to justify a decision-theoretic approach of an "inductive behaviour", but a single p-value does not tell us anything useful about a particular case.

Your "experiment" nicely demonstrates that small p-values do not "demontrate" that the null is false, and that large p-values also do not "demonstrate" that the null is true. It further shows that it makes no sense to compare p-values.

The results are correct. What may be wrong is your interpretation of what p-values are or what they would mean.

This is exactly what tests do: under the null-hypothesis they produce a random number, uniformly distributed in the interval [0,1] (= the p-value). The distribution of the p-values becomes right-skewed when the null hypothesis is false. The bigger the difference between the data and the null hypothesis, the more skewed is the distribution (this can be expressed as the "power" of the test).

This is also the reason why a single p-value can never be seen as evidence against the null hypothesis (or for the alternative). A samll p-value is a small p-value. What would be interesting is if this small p-value comes from a uniform distribution (the case where the null hypothesis is true) or not. There is nothing what can tell us this. Only the likelihood that the small p-value comes from a uniform distribution is lower than the likelihood that it comed from a right-skewed distribution. Therefore, in the long run, when repeatedly the null hypotheses are true, we would expect only fwe small p-values. Thus, p-values may be used to justify a decision-theoretic approach of an "inductive behaviour", but a single p-value does not tell us anything useful about a particular case.

Your "experiment" nicely demonstrates that small p-values do not "demontrate" that the null is false, and that large p-values also do not "demonstrate" that the null is true. It further shows that it makes no sense to compare p-values.

Thank you for this explanation, it cleared up a lot for me. However I am still unsure how to interpret some aspects of statistical testing.

When we are given some specific datasets to compare, we can only obtain one p-value (as opposed to synthetic datasets that I used, where I could repeat the tests and see that I indeed obtain a random number drawn from uniform distribution). Does that mean that, when rejecting H0, we accept the risk that the H0 may actually be true and we got the unluckily low p-value?

Also, do you mean that it doesn't make sense to compare p-values obtained using different tests? Or something else?

Does that mean that, when rejecting H0, we accept the risk that the H0 may actually be true and we got the unluckily low p-value?

Yes.

Also, do you mean that it doesn't make sense to compare p-values obtained using different tests? Or something else?

Example: a researcher is interested in the effect of some drug and how it depends on a particular genotype or condition. The researcher tests the drug effect in both genotypes or conditions. HShe finds the effect in one is "not significant" and in the other it is "significant" (or vice versa - does not matter). The conclusion that the drug effect is *different* for the two genotypes is NOT following from these p-values. Even if the drug effect was (non-)significant in both genotypes, it is NOT a valid conclusion that the genotype with the lower (higher) p-value was more (less) sensitive to the drug treatment.

That is exactly what I was looking for. Thank you for explaining this in a simple and lucid way.

One other thing to consider....

You are looking at random noise.

So I was interested in the difference between Cotton aphid biotypes. There is a yellow and a green biotype. I measured body length, and then did a randomization test for the difference in body lengths. If I did 10 randomizations, the p-value would jump around each time I did another 10 randomizations. So I did 100. the p-value was still all over the place each time I did another 100. I did, sets of 1000, then 10,000, and finally a few at 100,000. Each time the variability in the estimated p-value declined.

You have compounded the problem because you used the difference between two sets of random numbers, and only used 100 random numbers. There is a non-zero probability that you could get 100 zero values in one "treatment" and 100 ones in the other. If you increase the number of random numbers things will settle down.

This experiment has three obvious parameters that you could test.

1) The number of times you perform a test.

2) The number of values obtained from the random number generator.

3) The difference between sample 1 and 2. In this example the difference was zero, but you could add some small value to sample 2. What happens to the probability that I will get a significant difference if I add 0.01 to sample 2? How about 0.05, 0.1, 0.5, or 1?

4) If you have real ambition, what happens if the distributions are not normal?

Finally, Jochen's example is great. His final observation is clearly illustrated in your attached figure. Think of each call to the random number generator as a replicate. You then performed the test 100 times. In most of science the test is performed once. So now take one value from your figure, and that is the outcome for your test. You publish the result. Your peers do other research and they too publish their result from a single test. You now have a library filled with these experiments. 

Satendra -

You asked "Also, do you mean that it doesn't make sense to compare p-values obtained using different tests?" I noticed that you had this in your code:

p = kruskalwallis([sample1, sample2], [], 'off');

p = ranksum(sample1, sample2);

p = signrank(sample1, sample2);

So I assume you might have also meant that you expected different tests to likely have comparable p-values when an alternative hypothesis is 'true,' perhaps? If that is what you meant, then, let's say that there are different sensitivities involved in different tests. Here, the signed rank and rank sum tests work differently, and Kruskal Wallis is for more then two samples.

Attached you can get to something on a power analysis for the Wilcoxon Rank Sum test, if that is of any interest.

Also, you said "I was getting some unexpected p-values when performing statistical tests in Matlab (such as very small p = 1e-40) ...." That sounds like it was likely that either your sample size was extremely large, or the null hypothesis was far from true, or some combination of those was likely.

I think these may be the things that made you want to investigate in the first place.  Perhaps I have misinterpreted, and if so, I apologize for any confusion. 

Jim  

Conference Paper On the Lehmann power analysis for the Wilcoxon rank sum test

Article Practical Interpretation of Hypothesis Tests - letter to the...

From my experience I would especially agree with Ebert. Determining a proper (adequate) sample size can be very critical to achieving a meaningful result, particularly when differences are not large or a lot of random noise is present.