Dear Ronald,

If I understand your question correctly you are asking about the relationship between Level of Significance (LoS) and effect on an outcome. I think this question is rather too general to answer properly as it depends what kind of effect you are talking about and what the design of the analysis is.

I agree with Michael that correlation is not necessarily and indicator of causality. Also, sample size can have a big impact on the LoS in a correlation analysis independent of the effect (which would here be the size of the regression coefficient).

Generally speaking, if you take into account sample size and specify the design you can find a relationship between the LoS and the effect size. I would recommend googling for G*Power and having a look at their manual.

This is a very tricky business - strong positive correlation does not necessarily have anything to do with causality. Have a look at this very amusing - and enlightening - web page: 

http://tylervigen.com/spurious-correlations

I believe rejecting the null of zero effect is much more conclusive at the 0.05 level and the making a point of the practical or economic significance are all one needs. To be more exhaustive, assuming data availability, performance on "a hold-out sample" should make the issue exhaustive. I I hope this helps!

Dear Ronald,

If I understand your question correctly you are asking about the relationship between Level of Significance (LoS) and effect on an outcome. I think this question is rather too general to answer properly as it depends what kind of effect you are talking about and what the design of the analysis is.

I agree with Michael that correlation is not necessarily and indicator of causality. Also, sample size can have a big impact on the LoS in a correlation analysis independent of the effect (which would here be the size of the regression coefficient).

Generally speaking, if you take into account sample size and specify the design you can find a relationship between the LoS and the effect size. I would recommend googling for G*Power and having a look at their manual.

Peter's first and second paragraphs, mine suggestion and those of Michael arose because we are all approaching your question from different angles. From a non-econometric posture, effect size can be determined in behavioral sciences. My answer took both the econometrics and behavior into account. What constitutes a meaningful difference, as opposed to a a merely statistically significant difference? To answer such a question, we need a measure of the practical or clinical importance of an effect Such measure is referred to as the effect size. The criteria for judging such are necessarily subjective. That is, there are no tables of critical values for effect size as there are in hypothesis testing. However, there are guidelines by statistician on what is small, medium or large. See Cohen'd (1988). Small effect =.4 or less

medium effect=.5 to .7 and large effect is .8 or more.

Effect size for r can be obtained as square root as the (t squared /t squared + df). Or

Cohen d can be calculated to test for 2 sample mean as difference between the means divided by difference between the standard deviation of the dependent variables. From the t test we will know the p-value and from Cohen-d we will know the effect size. 

Ronald -

Peter mentioned power.  A type II error analysis is a fairly direct way to properly complement a p-value (which has little to no value by itself).   Together they cover what you would want to learn about effect size. 

 https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS

The American Statistical Association  (ASA) has finally taken a position on this:

 http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf

Cheers - Jim

Article Practical Interpretation of Hypothesis Tests - letter to the...

While I agree with James article concerning the p-value interpretation, I do not agree with his assertion that  American Statistical Association (ASA) of which I am a member has taken a position. 

While I agree with James article concerning the p-value interpretation, I do not agree with his assertion that  American Statistical Association (ASA) of which I am a member has taken a position- they gave guidelines

Dear All, 

Thank you for your genuine comments on this rather subtle topic!

To https://www.researchgate.net/profile/James_Knaub, thanks for pointing to the ASA's statement on the p-values, I also accessed:  Ronald L. Wasserstein & Nicole A. Lazar (2016): The ASA's statement on p-values: context, process, and purpose, The American Statistician, DOI: 10.1080/00031305.2016.1154108

The statement, however, falls short on providing guidance on the substantiality effect!

To https://www.researchgate.net/profile/Chuck_Arize, does Cohen'd (1988)  pre-suppose that effect sizes are only for continuous outcomes? 

If we assume a reasonably representative sample size, why wouldn't we conclude that the effect is substantial as long as the p-value is less than the set alpha-level?

For example, if the election guidelines of an election aren't followed or there exist some irregularities (in either cases p-value

No to Cohen_ d 

The second question -- we can if  by the definition of reasonably representative" one implies exhaustively representative.

Ronald -

You noted the following: "The statement, however, falls short on providing guidance on the substantiality effect!"

That comes from the type II error analysis. 

In a sequential hypothesis test you can directly see the difference between two "simple" hypotheses being compared.  Such a difference could be considered an effect, in my opinion.  This, I contend, is analogous to a confidence interval. 

The following may also be of help with regard to your question, as stated:

 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/

However, I cannot say that I have gone through it and can vouch for it, except to say that I have noticed generally good resource material from the NIH (US National Institutes of Health) ... as well as Pennsylvania State University, for that matter ... online. 

Cheers - Jim