Hi,
Considering the student's t-test formula, when sample sizes are large, it's easier to find significant p-values when they're not really there (i.e., false positives), because significance gets exaggerated. In other words, when one increase the sample sizes, any tiny effect will be significant eventually. In this situation, how to interpret the results of meta-analysis?
Any advise would be much appreciated.
You may check some latest discussions on this subject. e.g. Moving to a World Beyond “p
This is precisely why we encourage use of effect sizes. Whatever effect size version you choose (we can debate about that), the common characteristic is that it is not determined by sample size.
John
Peyman -
If you can use a confidence interval, I suggest you do that.
Hypothesis tests are often considered a way of making a decision, but without a power analysis, effect size study or other sensitivity analysis, and adjusting a threshold accordingly, one can see that sample size differences (depending upon standard deviation) make the idea of an hypothesis test as a decision making tool no more useful than a confidence interval, where available, and messier to interpret. Long ago, I developed an hypothesis test with computer support from colleagues, which dealt with sets of distributions, and we developed alternative hypotheses to show what results could mean, but often, with hypothesis tests, a confidence interval is a viable alternative, and is a useful tool for decision making.
In the past, the biggest problem with hypothesis tests seemed to be in small sample cases, where p-values tended to be large. Now with large data sets more common, people notice very small p-values. In both cases a threshold of 0.05 was often what people wanted to use. But what this really points out in both cases is the misuse of hypothesis tests, thinking that 0.05 or 0.01 were magic thresholds and no type II error analysis was needed. I think big data has made this problem more commonly noted, but I did publish a letter in The American Statistician in 1987 where I tried to make clear that hypothesis testing was not being done in a practical manner. It does take more work than publishing a p-value.
In your case, you seem to be able to use a confidence interval, so I suggest you do that. Perhaps publishing more than one confidence interval would be instructive. Relative standard errors are sometimes helpful. For regression, you can use prediction intervals. The estimated variance of the prediction error is often helpful. If the applicable distribution for a confidence or a prediction interval is oddly shaped, graphics might help. (Graphics are often helpful anyway.)
Cheers - Jim
PS - For big data, be sure the whole population is represented, not just all or most of one part.
James R Knaub I am not sure that confidence intervals (CIs) can help much in this case where p value does not help because there is a correlation (or equivalence) between hypothesis testing and CI. Basic and Applied Social Psychology (BASP) (an international journal) has banned CIs and hypothesis testing since 2015 (Trafimow and Marks 2015). The editors of BASP stated, “…confidence intervals do not provide a strong case for concluding that the population parameter of interest is likely to be within the stated interval. Therefore, confidence intervals also are banned from BASP (Trafimow and Marks 2015).”
Trafimow D and Marks M 2015 Editorial Basic and Applied Social Psychology 37(1) 1-2
Thank you all for kind suggestions,
Dear James,
Since the sample size affect confidence interval, I am not sure it is a good choice for addressing this problem. I need something unrelated to sample size.
Dear John,
Effect size would be appropriate, but can I consider a range between 0.2 and 0.5 as moderate effect?
Dear Hening,
I will get some information about SCI and if it is applicable I will give it a try.
You must decide on what represents an ES that is "moderate," or any other level. For instance, Cohen offered ES levels that have been useful, but although these are often used, it is specific to your research area what effect you would "salute."
How come your sample size is so large? Perhaps you should narrow down your
key words in your search?
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Sampling Design
- Sample Size