I recommend having them read the attached article

The point of confusion might be about the fact that a confidence interval has to be of something. Most commonly we calculate the confidence interval of the mean. Surely, the confidence interval for the mean shouldn't contain all the observed values in the sample ?

Now I'm stuck thinking about what the confidence interval for observations would be...

Hussain Ali , if you are talking about a 95% CI for the population mean, then why not just demonstrate with a simple example that many of the data points fall outside of the CI? E.g., I just generated 25 values using round(rnormal(50,10)) in Stata, then computed the 95% CI. Only 11 of the 25 values (44%) fall within the 95% CI. See the attached graph. If this is not what you are talking about, please clarify.

If we use the example of the CI for the population mean: You may argue that the estimate should become "better" (more precise) when more data (information) is available. So the expected width of the CI should decrease with sample size. By choosing an arbitrary large sample size you can get arbitrarily small expected CIs. But the sample size has no effect on the variance of the data itself.

A confidence interval is the mean of your estimate plus and minus the variation in that estimate. This is the range of values you expect your estimate to fall within a certain level of confidence for numerical data, but it is valid for data in a interval and ratio scale.

There is a specific type of confidence interval that may come close to having the property stated. This is often given it own name of “prediction interval”. Formally it is the same as a confidence interval for what a future single observation might be. As other have written, it is important to understand the target of the confidence interval.

In the case of the usual normal-distribution theory, it is easy to see what goes on. You might like to look at a situation where you are estimating both mean and variance from only two or three observations, but it is clear that in very large samples a 95% prediction interval will cover around 95% of the already-observed sample values.

In more practical situations you may still want to get a “prediction interval” for future observations. There are at least two difficulties in doings this: (1) you might be unsure of what model to use anf its number of parameters; (2) you might not have samples from the same environment as future samples. And you would need to think about the purpose for which a prediction interval would be used. In such situations you might need to drop the pretence of being able to do a formal analysis, and to drop the “percentage coverage” idea entirely, and to resort to some informal statement such as “it is likely to future observations will lie in this range” ... such an informal prediction interval would fairly naturally cover the range of all existing observations, but perhaps with same extra informal allowances.

Overall, it would be best if people didn't arbitrarily use the term "confidence interval" unless what they are doing is backed by some justifiable theory which accords with the standard basis of confidence intervals. Just adding a band to an estimate isn't good enough. A proper understanding of the meaning of the term is necessary.

Good point, David A. Jones. Your post reminded me of an appendix I included in a set of slides reviewing simple linear regression--see attached. There, the context is confidence intervals around a regression line in a scatter-plot. Some of the details may be a bit different when the context is a CI for a mean (e.g., the critical t might have df = n-1 rather than n-2?), but I think these slides might still be helpful.

Cheers,

Bruce

Dear Gi-Mick Wu , it's important to know that your "successful explanations" refer to the case that the population mean is already known. The ranges "+/- 2SD" and "+/- 2SE" are around the (usually unknown!) population mean. I am sure most people think these intervals are around the sample mean. But then your statements are not correct.

*Example: consider a random sample of size n = 3, from a normal distribution with µ = 120 and σ = 30: 54, 118, 82. The sample mean is m = 85, the sample standard deviation is s = 32. Correct: the probability of a new value being in the interval 120 +/- 60 is 95%, the probability of a new n=3 sample mean being in the interval 120 +/- 35 is 95%. However, following statements are wrong: the probability of a new value being in the interval 85 +/- 64 is 95%, the probability of a new n=3 sample mean being in the interval 85 +/- 37 is 95%.

The problem is, that we usually don't know µ and σ; we only know m and s. We can use m ans s (and n) to construct random intervals that have a defined probability of including µ. But this is not the same as the probability of µ being in a particular (realized) interval. You can easily see that when you understand that two independent samples from the very same distribution may have non-overlapping 95% confidence intervals. If "95% confidence" meant "95% probability of means falling inside the interval", then there had to be a 190% probability that a mean falls into either of the intervals.

Another way to understand that this interpretation is wrong is this: confidence intervals are random intervals with a defined coverage probability. Any rule adhering to this will produce valid confidence intervals. A simple possible rule to construct 95% confidence intervals is the following: toss a 20-sided die marked that das one black side. If the black size comes up, return an empty interval, otherwise return an open interval (-Inf, +Inf). Such intervals have the specified confidence, but it is obvious that an empty interval will have 0% probability of including µ, and an open interval will have 100% probability of including µ. It's also obvious that such confidence intervals are worthless, despite being correct. Good confidence intervals should have a finite span that is, in expectation, as small as possible.

Dear Jochen Wilhelm, indeed, it is for when the sample mean is equal to the population mean! So we should include the uncertainty about the estimated mean into the prediction interval. Or we go bayesian, which would be more in line with how most people interpret confidence intervals?

For the record: no, adding the probabilities of two confidence intervals from two samples to get 190% does not make sense at all even if the sample means were exactly equal to the population mean, so I think that's another issue, is it?

What is the point in providing a measure of uncerainty when you know the population mean? That is nonsense, even from a Bayesian perspective.*

And if the sample means are identical to the population means, then there is no way to get non-overlapping confidence intervals (at least not when they are constructed in the usual or at least some meaningful/helpful) way. Either I simply don't get your point or this is all nonsense*

*no offence intended! I only what to express that I don't see that your statements make any sense to me.

@ Hussain Ali

There are thousands of "non-statisticians" AND statisticians who do not know the meaning and the significance of Confidence Intervals.

There is no chance to teach them correctly...

OK, I stand corrected (and deleted the previous post to avoid confusion).

Here's now what I find the best explanation (link below), and it also explains the bayesian credible interval, which is how most people in my experience want from the confidence interval.

https://staff.math.su.se/hoehle/blog/2017/06/22/interpretcis.html

And further useful insights from Frank Harrell: https://discourse.datamethods.org/t/language-for-communicating-frequentist-results-about-treatment-effects/934