From what I have learned, when conducting subgroup moderator analyses in meta analysis (Hunter-Schmidt), the confidence intervals may overlap slightly but there may still be significant differences. Does anyone know of any references I can use to back up this claim?
My question is why you would want to back it up. Overlapping CI will nevertheless mean that the two estimates are not very robustly different (the p will be close to .05 anyway) and you will have a fairly high chance the difference does not replicate as well as one would hope. I would also recommend trying to understand whether overlapping CI are mainly due to considerable variation in either (or both) subgroups, or whether it is more likely due to lack of power. I think this will provide more valuable insights than the p-value itself.
Hello Alex,
Here's an example showing the phenomenon you described:
Let A and B represent two independent groups (each with n = 50 cases).
Mean(A) = 54, SD(A) = 10
Mean(B) = 50, SD(B) = 10
95% CI for population means:
Group A: (51.23, 56.77)
Group B: (47.23, 52.77)
So, the CIs overlap.
T-test for independent groups (A, B): t(98) = 2.00, p = .048 (non-directional).
So, yes, the difference between observed means is statistically significant.
Note that requiring the 95% CIs not to be overlapping in order to assert differences is akin to saying the observed means must be approximately 4 standard errors apart (using individual group SE values), whereas the ordinary t-test at the .05 level only requires that observed means be approximately 2 standard errors of the difference apart. Here, SE(diff) = 1.41*SE(individual batch) (when group variances and sample sizes are equal). So, the group means only need differ by about 2.8 SEs (individual batch) for statistical significance in the t-test, not the approximate 4 SEs required for the nonoverlapping CI criterion.
Good luck with your work!
David Morse
Hi David, Thank you so much for your response. This is exactly what I was looking for. In a number of cases, my confidence intervals overlap minimally. To say that the groups aren't different because there is such slight overlap would seem incorrect. Do you know of any articles I can cite that illustrate what you have shown above? Thanks again!
Hi Jan Antfolk
Thanks for your response. The reason I was asking is because in many cases there is very minor overlap in confidence intervals. So it seemed odd to say that the groups weren't different from one another.
Hi Alex,
this paper might be what you're looking for:
Cumming, G., & Finch, S. (2005). Inference by eye: confidence intervals and how to read pictures of data. American Psychologist, 60(2), 170
Good luck!
- Badges
- Science method
- Similar topics
- Mathematics
- Statistics
- Statistical Techniques
- Meta-Analysis