Judging the significance of a mean difference from the overlap of CIs of group means can be very misleading, particularily when the values in the groups are not independent.

Article The Correct Interpretation of Confidence Intervals

https://towardsdatascience.com/why-overlapping-confidence-intervals-mean-nothing-about-statistical-significance-48360559900a

https://www.cscu.cornell.edu/news/statnews/stnews73.pdf

Article A brief note on overlapping confidence intervals

Thank you Jochen Wilhelm . Very informative paper.

Here's another (very short) article to add to Jochen's nice list. HTH.

Article If we're so different, why do we keep overlapping? When 1 pl...

Adding another ref: http://www.bristol.ac.uk/media-library/sites/cmm/migrated/documents/graphical-presentation-of-a-collection-of-means.pdf

Here the example is when you have several objects to compare.

The school scores Example in Daniel's link is interesting...

The estimates across the schools are pretty well normally distributed. This indicates that these estimates may be seen as a random sample from a normal distributed population of schools. The analysis thus lacks "school" as a random factor. I wonder what then is the point is asking for "significant differences" between schools? The data per school may be sufficient to declare a difference in their scores, but as the scores per scool are just varying (without knowing the reasons for this variance), there is no relevant information associated with this insight.

If one had an "experimental factor" (like e.g. the average class sizes, or teaching concepts, or playground structure etc.) that was known for the schools, one might ask the really meaningful question what impact such a factor had on the scores. But that would result in a regression analysis in which one might test the coefficient coding the effect of that factor - the comparison of CIs for two selected schools is not required at all, and there is no mutliple testing with all its problems.

Goldstein has done a lot of work treating school values as a random normal sample (e.g., https://www.amazon.com/Multilevel-Statistical-Models-Harvey-Goldstein/dp/0470748656 ). These plots are often called caterpillar plots and are used in much school effectiveness reporting (and other areas). Usually they are reported as the intervals (different types) from random coef models using the shrunken estimates. So in R:

I agree in most contexts you wouldn't be comparing pairs. In this particular paper they discuss the situation where a parent wants to compare two schools. They have done on many more issues on problems with people trying to use data like these from making choices between schools.

Thank you, Bruce Weaver . This is a very informative and didactic paper. I therefore believe that the use of CI's will be discriminatory only for p < 0.01.

Just to clarify. For independent normal means you have equal n, equal variance CIs then the 95% CIs for each mean too wide for non-overlapping CIs to correspond to the p < .05 threshold for a difference in means. To adjust to get the right width you need to plot 2^0.5 * tcrit SEs or the interval or 0.5 * 2^0.5 * tcrit for the error bars (half width). As tcrit is about 2 in most cases this will require putting approximately 2^0.5 or 1.41 SEs in most cases. Goldstein and Healy (working generally with large samples) derive the value 1.96 * 2^0.5 = 1.39 and reccomend plotting 1.4 SEs in their caterpillar plots.

If you don't plot corrected error bars rule of thumb is the CIs can overlap around a third of an error bar and be just statistically significant. Error bars that don't overlap will be statistically at somewhat below p < .05.

This all assumes the CIs are derived by a reasonable procedure for you data, means are independent and SEs broadly similar between means being compared.

See also Article Calculating and graphing within-subject confidence intervals for ANOVA

Thanks, Thom Baguley . Very clarifying. Your paper is also very informative, with R analysis. Great!