Theoretically, when the sample size tends to be infinite, any association, either a correlation coefficient or a difference, will be statistically significant. Some researchers report p values very small to show rigor of their findings. But how much small p value is enough to make a finding rigor? Is there any relationship with the sample size? How about this suggestion came to my mind: extracting a small random sub-sample from our big sample and test the significance of association among them, not in the whole sample?
.
very true ; see the notion of "effect size"
https://en.wikipedia.org/wiki/Effect_size
(and the comment about the misuse of p-values)
if you are concerned by the smallness of the effect size with very large samples, you could just avoid testing against an equality null hypothesis ; you could test for a minimum difference between the samples, hence controlling directly the"pragmatically relevant difference" between samples (regardless the size of the samples) instead of probing the "statistically significant difference" (sample-size dependent as with an equality null hypothesis)
.
That is a very good question.
What comes to my mind is this: Go for literature review and see if effect sizes correspond with your results.
Well it is true that increasing sample sizes you increase a chance do detect significant associations. However, with underpowered studies you are facing the possibility of Type II errors to occur.
Regards, Witold
Hassan -
It has long been a problem that p-values have been misunderstood and overused. An isolated p-value is nearly meaningless. See my explanation of interpretation of hypothesis testing here:
https://www.researchgate.net/publication/262971440_Practical_Interpretation_of_Hypothesis_Tests_-_letter_to_the_editor_-_TAS
The American Statistical Association (ASA) has finally - way overdue - taken a stand on this. Here is a recent press release:
http://www.amstat.org/newsroom/pressreleases/P-ValueStatement.pdf
The basic fact is that a p-value is a function of sample size, like a standard error of a mean used in a confidence interval. But a confidence interval - or in regression, a prediction interval - gives you two end-bounds for a given level. A p-value is misleading because it only gives you one measure that leads people to make decisions that may be based on sample size alone. With a small enough sample size, you cannot 'disprove' anything. With a large enough sample size, nothing seems possible to be 'proven.' Thus effect size is important, as Fabrice indicated, and I think such considerations are often handled best by considering probabilities of type II errors, as noted by Witold.
Regarding effect size, this looks like it might be interesting for you:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3444174/
At any rate, i have often found that there are two sources which generally provide good information on statistics, online: (1) NIH (US National Institutes of Health), and (2) PSU (Pennsylvania State University). There are many other good sources, but these seem to be generally good, rather consistently.
Cheers -
Jim
PS - Andrew Gelman has an interesting note here, especially regarding large sample sizes:
http://andrewgelman.com/2009/06/18/the_sample_size/
Article Practical Interpretation of Hypothesis Tests - letter to the...
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size