The question somewhat interrogates the effects of statistical levels of significance (LoS) (0.01, 0.05, 0.1 etc..) in determining substantiality effect on an outcome. Can this be determined without conducting a simulation study? If not what would be the best initial guess or efficient algorithm to establish an appropriate LoS for the substantiality effect?
This is a very tricky business - strong positive correlation does not necessarily have anything to do with causality. Have a look at this very amusing - and enlightening - web page:
http://tylervigen.com/spurious-correlations
I believe rejecting the null of zero effect is much more conclusive at the 0.05 level and the making a point of the practical or economic significance are all one needs. To be more exhaustive, assuming data availability, performance on "a hold-out sample" should make the issue exhaustive. I I hope this helps!
Dear Ronald,
If I understand your question correctly you are asking about the relationship between Level of Significance (LoS) and effect on an outcome. I think this question is rather too general to answer properly as it depends what kind of effect you are talking about and what the design of the analysis is.
I agree with Michael that correlation is not necessarily and indicator of causality. Also, sample size can have a big impact on the LoS in a correlation analysis independent of the effect (which would here be the size of the regression coefficient).
Generally speaking, if you take into account sample size and specify the design you can find a relationship between the LoS and the effect size. I would recommend googling for G*Power and having a look at their manual.
Peter's first and second paragraphs, mine suggestion and those of Michael arose because we are all approaching your question from different angles. From a non-econometric posture, effect size can be determined in behavioral sciences. My answer took both the econometrics and behavior into account. What constitutes a meaningful difference, as opposed to a a merely statistically significant difference? To answer such a question, we need a measure of the practical or clinical importance of an effect Such measure is referred to as the effect size. The criteria for judging such are necessarily subjective. That is, there are no tables of critical values for effect size as there are in hypothesis testing. However, there are guidelines by statistician on what is small, medium or large. See Cohen'd (1988). Small effect =.4 or less
medium effect=.5 to .7 and large effect is .8 or more.
Effect size for r can be obtained as square root as the (t squared /t squared + df). Or
Cohen d can be calculated to test for 2 sample mean as difference between the means divided by difference between the standard deviation of the dependent variables. From the t test we will know the p-value and from Cohen-d we will know the effect size.
Ronald -
Peter mentioned power. A type II error analysis is a fairly direct way to properly complement a p-value (which has little to no value by itself). Together they cover what you would want to learn about effect size.
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
The American Statistical Association (ASA) has finally taken a position on this:
http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
Cheers - Jim
Article Practical Interpretation of Hypothesis Tests - letter to the...
While I agree with James article concerning the p-value interpretation, I do not agree with his assertion that American Statistical Association (ASA) of which I am a member has taken a position.
While I agree with James article concerning the p-value interpretation, I do not agree with his assertion that American Statistical Association (ASA) of which I am a member has taken a position- they gave guidelines
Dear All,
Thank you for your genuine comments on this rather subtle topic!
To https://www.researchgate.net/profile/James_Knaub, thanks for pointing to the ASA's statement on the p-values, I also accessed: Ronald L. Wasserstein & Nicole A. Lazar (2016): The ASA's statement on p-values: context, process, and purpose, The American Statistician, DOI: 10.1080/00031305.2016.1154108
The statement, however, falls short on providing guidance on the substantiality effect!
To https://www.researchgate.net/profile/Chuck_Arize, does Cohen'd (1988) pre-suppose that effect sizes are only for continuous outcomes?
If we assume a reasonably representative sample size, why wouldn't we conclude that the effect is substantial as long as the p-value is less than the set alpha-level?
For example, if the election guidelines of an election aren't followed or there exist some irregularities (in either cases p-value
No to Cohen_ d
The second question -- we can if by the definition of reasonably representative" one implies exhaustively representative.
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis