I have read many articles encouraging the use of confidence intervals instead of post-hoc power analysis.
Saúl -
The key in both cases is estimation of standard error (assuming bias and nonsampling error are negligible - which may not be true), for example, the standard error for a mean. This is dependent upon inherent population standard deviation and sample size. We may 'guess' standard deviation based on preliminary information. For yes/no cases we may take a worst case. For various designs, beyond simple random sampling, there are complications. But the basic ingredient for both power and confidence studies is standard error from standard deviation and sample size. Confidence intervals, however, will often be more practically interpretable. You may just need an estimate of relative standard error: estimated standard error of perhaps a mean (or total), divided by the estimated mean (or total).
Cheers - Jim
Post-hoc power analyses are nonsense, and they can't be used to critizise the conclusions of an article (they can't be used for anything).
Just because one can use a hammer to hit a chump that contains nails does not mean that this action is somehow sensible.
If you are able to meaningfully compare probabilities for alternative hypotheses, that might sometimes be desirable, but I have not had need for a hypothesis test in many years (much less "significance," which is a misleading term), and think this generally overused and misused. - I do remember sequential hypothesis testing to be an interesting way to decide between two 'simple' alternatives, which employs a variable sample size.
--------------------
Two references:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
https://www.researchgate.net/publication/271072864_Analyzing_n_Samples_of_2_Observations_Each
You will see a link to a proceedings containing that paper.
Article Practical Interpretation of Hypothesis Tests - letter to the...
Conference Paper Analyzing n Samples of 2 Observations Each
Dear All,
Jochen is simply right: http://daniellakens.blogspot.hu/2014/12/observed-power-and-what-to-do-if-your.html
Dear Sirs,
Thanks for your replies. I have read in well-cited papers that confidence intervals are better to inform readers about the possibility of an inadequate sample size than do post hoc power calculations. See:
https://www.ncbi.nlm.nih.gov/pubmed/11310512
http://www.tandfonline.com/doi/abs/10.1198/000313001300339897
I would like to know your opinion about reporting the interpretation the CI + another statistical descriptors.
Cheers
A CI is more useful if the estimate is important. A p-value is suffcient if only a test is important. The test is just to judge if one is sufficiently confident about the direction of the effect, but not about the size of the effect. Especially when the effect is relatively large or is estimated with high precision, a test (which would give a very tiny p-value) is not very instructive, but it might be interesting to see if (how well) the data would be compatible with a smaller or larger effect as well.
However, the CI is still just about the same thing than the p-value and simply shows the test result from a diffrent angle. And because it is about the test, it is about the data, not about the estimate. A direct statement about the estimate would require Bayesian statistics. But given a flat prior and a sufficiently large sample size the CI can serve as a sufficiently good approximation for the credible interval of the estimate.
A p-value does not stand alone well. What is "big"? What is "small"? Note that convention on that does not fit with "big data." A p-value in a vacuum is rather meaningless.
An interval is not one ended.
P-values and confidence intervals are both sample size dependent because standard errors are. But a p-value takes the standard error and converts it to something which is not interpretable by itself. That has caused a great deal of misuse.
http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#.WVO9FcspDqA
No.
Sample size is contextual both within a general framework of similar studies and within the study itself. I can run an experiment on perch with 10 replicates and be rightly criticized because perch are common fish. The same sample size on rare pelagic sharks might be astounding. A study that killed 60,000 black cutworms might be just fine, but I might have troubles if I used 60,000 baby rabbits.
My thought is this: If the sample size is in line with existing publications or substandard with extenuating circumstances [costs (time, or money), risk (bodily harm), rarity], then the study should be published. It is unfair to punish one research group for a failing of all. It is then our responsibility to write editorials to journals and point out the problems associated with small sample sizes in terms that the readers of those journals can relate to.
At some point every graduate student in the biological sciences should have to answer the questions posed to the ASA board in James' linked article. However, it begs to have the questions changed a bit to:
Q1) What do you think is an appropriate p-value for deciding to reject the null-hypothesis?
Q2) How did you make this determination, or more importantly how should others make a similar determination that fits their research project objectives?
Dear Timothy,
In my opinion, it is not a matter of "punish one research group for a failing of all" but of informing the reader that the results of the study fall by itself due to the insufficient sample (and therefore, statistical power). It is a responsibility as researchers to report failures that may affect the conclusions of a study and, therefore, other researchers on the subject.
In this case, there are no extenuating circumstances and the authors did not calculate a priori power so their results are underpowered.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics