I just want to second Alex' answer. And I also find the odd-ratio thing wrong, because this implies that (a) p is the probability of the null hypothesis (H0) and that (b) 1-p is the probability of the alternate hypothesis. Point (a) is wrong straight away, because p is the probability of the data, *given* H0. Point (b) is wrong because (a) is already wrong, and also because 1-p is not even simply the inverse probability (of *not* getting such data given H0). The odds of not getting such data is more related to the power or type-II error (beta).

I just want to second Alex' answer. And I also find the odd-ratio thing wrong, because this implies that (a) p is the probability of the null hypothesis (H0) and that (b) 1-p is the probability of the alternate hypothesis. Point (a) is wrong straight away, because p is the probability of the data, *given* H0. Point (b) is wrong because (a) is already wrong, and also because 1-p is not even simply the inverse probability (of *not* getting such data given H0). The odds of not getting such data is more related to the power or type-II error (beta).

Gentlemen, thank you for your thoughtful answers.

Sorry but I should have mentioned for this example that the assumption is Ho is false. As mentioned by Lombardi and Hurlbert (2009), the difference between means can be determined as (1-p/2) but also as an odds ratio (1-P/2)/(P2). Hence my example that if the result of a test between two means yields a p-value = 0.2 and (Mean A – Mean B)>0, then the odds are 9:1 that µA> µB. This is paraphrasing closely the text in Lombardi and Hurlbert (2009).

I should say the point here is not to replace the p-value but to provide interpretation. Every scientist understands that the result may be important if p

Thanks Alex for your detailed reply. Frankly I don’t think it’s a slippery slope. As I mentioned before, I believe it’s important to publish actual p-values and provide some narrative to facilitate interpretation so the reader can decide. The odds ratio may be a poor choice; however, as you aptly point out, additional information is necessary.

I am not saying low p-values (e.g., 0.05 – 0.2) are important, but I am implying that they should not be ignored. Also keep in mind that many researchers live and die by the p-value and I believe many do just look at that value and do not consider the ramifications. Frankly, I think the researchers that are “worth their salt” are in the minority. This is also important for outreach to the public and media and I think as scientists we should emphasize that there is nothing magical about p ≤ 0.05. This is also relevant for explaining how a low R2 may be useful.

From what I’ve learned, Fisher in his later years essentially agreed with Pearson that setting the significance level (alpha) was unnecessary and reporting exact p-values was better. So with that, how is one to interpret p-values without some indication of their importance? That was the genesis for my original query. Most of this, and much more is detailed by Hurlbert in his neoFisherian paradigm.

In almost every context a confidence interval is more informative than a significance test and provides a better summary of the information - for precisely the reason you give (the preponderance of low power studies). In the charts below, ( and ) represent lower and upper confidence intervals, `|' represent values which are in some sense biologically important, and 0 represents zero for an effect. The scale for a parsmeter is represented by ----

Case a: --|---{--0--)--|--

The parameter is not statistically significantly different from zero and is clearly not large enough to constitute an important effect

Case b: --(--|--0--|--)--

The parameter is not statistically significantly different from zero and may or may not represent an important effect

Case c --|--(--)--0---|--

The parameter is statistically significantly different from zero but does not represent a biologically important effect

Case d --{--|--)--0--|--

The parameter is statistically significantly different from zero and may or may not represent a biologically important effect

Case e --(--)--|--0--|--

The parameter is statistically significant and clearly does represent a biologically important effect.

So here are 5 situations with different implications being summarised as significant or not significant.

Arguably, for Bayesians like me, even confidence intervals a very bad at doing this job. Here is an old chestnut which explains why.

Consider a sample of size two drawn from a distribution which is uniform on alpha +/- 1. Now the interval min(X1,X2),max(X1,X2) is a 50% confidence interval for alpha. That is, it has a 50% probability of containing the true value of alpha irrespective of what alpha actually is.

But is |X2-X1|>1 the interval must contain alpha with probability 1. So here is a valid 50% confidence interval with the property that when we calculate it, we sometimes know it contains the true value.

This arises because, as Hacking says, confidence intervals are rules for before trial betting not after trial betting. A Bayesian credible interval does not have this property. Interesting;ly,. Fisher argued that we should always condition on ancillary statistics. In this example |X2-X1| is ancillary, and if we construct a confidence interval based on Fisher's approach we don't get this [problem. Another example of Fisher's wisdom and Neyman Pearson folly :)

Alex,

I think we do agree on many issues. That mchankins article is very entertaining, and I plan to use some of those gems. Unfortunately, this article perpetuates the concept of “significance”. All those terms describe their p-values in terms of significance, which is the problem. The conclusion is that results are significant or not and I think that’s wrong. Exact p-values are informative and the goal is to provide those and not couch them in terms of how they relate to the magical, and artificially generated bright line of “significance” that is supposed to indicate important from not important. As a biologist, I am not going to ignore p=0.06 or any other relatively low p-value. Of course, as you point out, it would get fuzzy when you get above, let’s say 0.15. In many cases the results from biological experiments with p-values in that range (e.g. 0.05 - 0.15) are important, given all the parameters that go into these values, as mentioned above by Glenn. I’m sure a good deal of this is discipline-dependent; hence a biologist and physicist would likely have very different views on the importance of p-values from statistical tests.

The follow-up to the mchankins article (http://mchankins.wordpress.com/) shows how people waffle when p-values are between 0.05 and 0.15, which aptly highlights the general discomfort surrounding the bright-line alpha!

I'll fire in some promotion for a close friend and colleague of mine here. Have you heard of the optimal-alpha approach to hypothesis testing derived by Joe Mudge and colleagues at the University of New Brunswick? Essentially what they're proposing is that we should be considering the relative weights of alpha and beta when defining thresholds by which to test for type 1 and type 2 error. In addition, we need to strongly consider the critical effect size that we want to detect in our studies in order to derive appropriate alpha and beta thresholds at which to test for significance. These alpha and beta levels of significance should not arbitrarily be defined, but need to be carefully thought out and derived from empirical data present within the scientific literature. If such a means of deriving an appropriate critical effect size isn't possible, we need to at least begin setting alpha and beta equal to minimize the chances of making ANY type of error in statistical hypothesis testing. The arbitrary cutoffs of 0.05 and 0.20 for alpha and beta, respectively, just don't have any evidential foundation and are, in my opinion, completely inadequate.

You can find their paper at the following link:

Mudge, JF, Baker, LF, Edge, CB, and Houlahan, JE. 2012. Setting an Optimal α That Minimizes Errors in Null Hypothesis Significance Tests. PLOS One. 7: e32734. DOI: 10.1371/journal.pone.0032734.

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0032734

"Statistical significance" has very little to do with "scientific significance". If you want to know what the data is telling you, a confidence interval e.g. with conventional confidence level 95%, is often very revealing. You are trying to measure something and the data tells you something about it. Maybe a lot, maybe almost nothing.

Hi folks,

I'm a big fan of confidence intervals that have been promoted for many years in the medical literature.  More recently I've been including raw data points with my confidence intervals (see my attached paper).  The Drummond and Vowler paper below highlights the value of showing the raw data.    

Below are a few links to some good resources by Geoff Cumming who has written a lot about confidence intervals (see book below) and made some brilliant videos too ! (see links below).  Also included a few of my other favorite confidence interval papers below. 

http://www.amazon.com/Understanding-The-New-Statistics-Meta-Analysis/dp/041587968X   

-Cumming, G., Fidler, F., Vaux, D.L., 2007. Error bars in experimental biology. Journal of Cell Biology 177, 7-11.

-Cumming, G., Finch, S., 2005. Inference by eye - Confidence intervals and how to read pictures of data. American Psychologist 60, 170-180.

Cohen J. 1990. Things I have learned (so far). Am. Psychol. 45:1304-1312.

Drummond G.B., S.L. Vowler. 2011. Show the data, don't conceal them. Clin. Exp. Pharmacol. Physiol. 38:287-289.

Beninger P.G., I. Boldina, S. Katsanevakis. 2012. Strengthening statistical usage in marine ecology. J. Exp. Mar. Biol. Ecol. 426:97-108. 

Lambdin C. 2012. Significance tests as sorcery: Science is empirical-significance tests are not. Theory & Psychology 22:67-90. 

https://www.youtube.com/watch?v=5OL1RqHrZQ8

https://www.youtube.com/watch?v=iJ4kqk3V8jQ

Article Agronomy of strip intercropping broccoli with alyssum for bi...

Agreed.

Hope this helps,

Matt