Hi Margarget,

You're absolutely correct.  When a p value is larger than the criterion, we cannot conclude that two distributions are not different, merely that we cannot detect a difference in the sample data.  This could be due to lack of power, sampling issues etc.  

If one wants to "argue the null", one needs to either conduct a Bayesian analysis (which if the null is likely would construct a distribution around zero).  

If one wanted to stick to frequentist statistics, one would need to conduct a power analysis to argue that the number of samples required to observe a difference between the two measures is so high as to be unlikely to ever be observed in nature.  

Hope this helps,

John

I think that those types of interpretations are routinely used in that way. For instance, in linear models (regression, ANOVA) the assumptions verification involves testing that the residuals distribution is not different from a normal distribution. If a p-value is above 0.05 (and the other assumptions stand) the linear model is considered valid, and their effects considered significant, instead of the more elaborate interpretation that "since departure of residuals from a normal distribution was not detected, either our effect is strongly significant or our sample was too small to detect violation of our assumptions". I think that in scientific disciplines that are very experimental and use intensively statistical inference, as biology (my field), it could be frequent that researchers unconsciously take a mental shortcut that makes "absence of evidence" equal to "evidence of absence".

On the other hand, I think it could be reasonable that a good knowledge of the system under study, altogether with the lack of evidence could serve to construct a good argumentation regarding the real absence of effect, or of an enough absence of effect. Unfortunately, I'm afraid that making the analysis suggested by John (Bayesian approaches, power analysis, and other warrants) are not as widespread as they should. In some cases, even assessment of violations of the assumptions is taken for granted in large scale studies.

Just an addition to the previous two posts:

There is no way ever to demonstrate that an effect is absolutely abscent.The frequentistic way of doing a power analysis and then obtaing a non-significant result only means that in this case one may claim that the effect is below a minimum relevant effect (that is not zero) while keeping a desired false-negative rate.

A good way to communicate the results to allow the audience to follow the argumentation is to show the confidene interval. It should include the zero and it should be narrow enough to not include relevant non-zero effects.

Clearly, simpliy giving a p-value, especially without stating the power of the test, is wrong and can not support the claim of having a zero (or a non-zero but negligible) effect.

PS: I know statistics professors who can not explain correctly what  a p-value is. And who do not know the difference between hypothesis testing and significance testing.

Even in large sampling situations, most standard tests will be only telling you whether the means of two distributions are (significantly) different, but nothing about how different they really are. You can have distributions whose means differ by 0.01 (just to give a number) and still be statistically different. The best option in this case is, of course, to plot and show your data.

In this regard, I think you may find interesting the paper "Not different is not the same as the same: how can we tell?", published in the Journal of Physiology (link attached). Other papers published in their special series of 'Statistical Perspectives' (http://jp.physoc.org/cgi/collection/stats_reporting) may be also of interest.

http://jp.physoc.org/content/590/21/5257.full

You are correct; the use of the p value in this way is simply wrong. Interestingly, the use of the p value to demonstrate evidence for a difference is also wrong. The p value has no connection with the strength of the evidence for a difference and it is easily demonstrated that it is logically invalid to use a significant p value to claim that the null hypothesis is unlikely.

The only thing a p value is good for (with reference to a null hypothesis, at least) for is making a decision with a Type I error rate of alpha. It cannot be used to argue for *or against* the null hypothesis. If you want to do that, use Bayesian statistics.

There are quite a few issues here. First, the inference would, as others have said, have been much better stated in terms such as "not significantly different" rather than "the same"; the latter is at best a loose shorthand for the former and at worst an incorrect conclusion.

Second, the use of p>0.10 is not a particularly common criterion; 0.05 is probably the most common. However, while using 0.10 will (by definition) result in a higher rejection rate at the null, it also increases the test power for the alternate hypothesis, in effect providing a somewhat more sensitive check for a difference than usual. This does add a little more weight to the conclusion that the true difference is small.

Third, the general issue that a report of statistical significance, by itself,  says nothing about the observed size of the effect,  or about whether the effect is (or could be) large enough to worry about or small enough to detect. As ever, the latter is usually a much more important question and is better answered by reporting the estimate with something like a confidence interval.

Finally, to get back to the conclusion of no difference, whether it is reasonable to conclude that the difference is indeed zero depends not only on the experimental evidence but on the prior probability of the difference being zero. In most real cases, the chances of two appreciably different treatments having _identical_ effects are pretty much zero -  if the treatments are sufficiently different to be interested in measuring them, there _will_ surely be _some_ difference in effect, though it can of course be small. And with a zero prior probability for zero difference, no amount of evidence can lead to a nonzero posterior probability of zero (true) difference (I've obviously shifted into Bayesian thinking here). So in practice, pretty much any conclusion that two usefully different treatments have exactly the same true effect is going to be wrong, no matter what a p-value says.

To add to the other comments. The valid way to provide evidence of absence of an effect in a frequentist setting is an equivalence/noninferioirty test. you basically turn round the ususal hypothesis and alternative, but there are subtleties like the non-inferiority margin you need to specify. Also in this setting you can end up with an inconclusive situation, i.e., that you cannot reject a relevant difference.

Hope that was helpful

Me as devil's advocate:

The questioner does not say what the statistical test is, so let's say it is something about the shape, location, and spread of the distribution, and let's assume the sample is large, and that speaker did not mean exactly the same, but close enough.

Again, let's suppose that this was a key part of the speaker's thesis. He or she put forward the bold conjecture that the two distributions should be the same, and put it to a stringent test with a large data set. This seems good Popperian science. You are supposed to put forward bold conjectures and test them.  Okay, the speaker could say that her/his theory was not refuted/falsified by the data, but I wasn't there so don't know the exact language used.

For me I would like to know more details before judging the speaker. Of course there are the general arguments against using p values for anything, but I am assuming here that people don't want to get into that debate.

I will participate with one example to discuss:

Supose that you have two different methods to measure the same variable and conduces a randomized paired design with 200 samples, the mean of the difference between methods ± st deviation was -0.1±2.02, 95% CI for the difference -0.38 to + 0.18, p value=0.4852 (t paired test) and concordance correlation coefficient (Lin) = 0.9968

¿What is your conclusion?

My conclusion is that the difference between these methods is unlikely to be greater than 0.4 (in either direction). If this 0.4 is negligible for the prurpose, the following conclusioon was that it does not matter which method I chose (i.e. one can simply decide for the cheaper, simpler, more convenient method). If a difference like 0.2 was relevant, the conclusion was that the data is not sufficient (not precise enough) to support a decision on which method to use.

Thank you Jochen, it's precisely my point, you use the statistic evidence for take decisions in your work and you must be helped by other kind of knowledge that gives to you the sight to interpret the statistic results in the better way. Your answer reflects that situation.

Statistics must be understood for practical purposes and we should not discuss too much about what could be significant for me or someone else, the researchers should use the statistical answers in terms of the objectives of their research helped by non statistical knowledge of the problem.

p value coul be considered old and unuseful, but when  confidence interval and the p value meaning the same, you think it is not to bad thinking that p value, it's not to old and unuseful.