Dear Hening Huang,

Thanks for the nice discussion. I would add that Confidence Intervals are also often misinterpreted when used for design purpose, like in flood design. In this context, I think it is worth mentioning the paper by Serinaldi and Kilsby (2015):

‘Since hydrological frequency analyses are usually based on short time series such as a few tens of annual maxima or slightly longer sequences of peaks over threshold, the sampling and estimation uncertainties are commonly large. For instance the 95% CI of the probability of exceedance of the largest observation in a sample of size 50, which is estimated as 1/(50+1)=0.0196 according to the Weibull probability unbiased estimator, ranges from 0.0005 to 0.071 [38]. In other words, we can only say that there is a 95% probability that the interval between about fourteen and two thousand years contains the ‘‘true’’ return period of our ‘‘observed 50-year event’’. ‘‘Anything beyond this kind of specification is speculation, notwithstanding any mathematical legerdemain by which it could have been obtained’’ [38].’

Serinaldi, F. and Kilsby, C. G. (2015) Stationarity is undead: Uncertainty dominates the distribution of extremes. Advances in Water Resources, 77, 17–36.

[38] Klemeš V. Risk analysis: the unbearable cleverness of bluffing. In: Bogardi JJ, Kundzewicz ZW, editors. Risk, reliability, uncertainty, and robustness of water resource systems. International hydrology series. Cambridge University Press; 2002. p. 22–9. http://dx.doi.org/10.1017/ CBO9780511546006.003.

Dear Baptiste Francois

Thanks for your comment and input. I would like to add a note from Deming. In his book entitled “Out of the Crisis,” Deming (1982) stated,

“There are many other books on so-called quality control. Each book has something good in it, and nearly every author is a friend and colleagues of mine. Most of the books nevertheless contain bear traps, such as reject limits, …areas under the normal curve, acceptance sampling….The students should also avoid passages in books that treat confidence intervals and tests of significance, as such calculations have no application in analytic problems in science and industry.”

Apparently, Deming’s warning has been ignored.

It is important to distinct Neyman’s CI from probability interval (PI). The physical meaning of CI is very different from the physical meaning of PI.

I used the word ‘stick’ to describe the t-interval for μ (x ̅- ts/√n, x ̅+ts/√n), a well-known CI (Huang 2018). This stick notion helps to understand the physical meaning of the t-interval or CI procedure. Let us use ‘sticks’ to conduct a ‘physical experiment’. Step 1, draw a line on the ground, denoted by L-L. This L-L line represents the true value μ. Step 2, collect a large number of wood sticks to represent the realized t-intervals or CIs. Step 3, throw the wood sticks onto the L-L line. Step 4, calculate the frequency that the wood sticks have captured the L-L line. To reflect the physical meaning of this t-interval stick procedure (or any CI procedure), we may designate this frequency as ‘capture rate’. The mathematical expression of the capture rate associated with the t-interval is written as

Capture rate=Pr([x ̅- ts/√n, x ̅+ ts/√n] ∋ μ)=1-α

The ‘capture rate’ 1-α is known as ‘confidence level’, ‘coverage probability’, or ‘long-run success rate’ in the literature. However, among these four terms, ‘capture rate’ most accurately reflects the physical meaning of a CI stick procedure. Note that capture rate is a long-run property and is associated with a CI stick procedure only. For a realized CI stick from a sample at hand, the capture rate, i.e. the probability with which the unknown μ is captured by the realized CI stick, is either 0 or 1. Therefore, a CI stick does not describe measurement uncertainty.

On the other hand, a well-known PI is the z-interval for measurement error ε: (-zσ/√n, +zσ/√n ). Note that the z-interval is a fixed interval with a random subject ε. Its physical meaning is that, if we make a large number of measurements, 1-α percent of the measurement errors will fall into the fixed interval (-zσ/√n, +zσ/√n ). In other words, for a measurement we have made, we are 1-α percent certain that its measurement error ε is within the z-interval (zσ/√n, +zσ/√n). In order to reflect the physical meaning of the z-interval, we may designate the probability 1-α as the degree of certainty, which is expressed as

Degree of certainty=Pr(ε ∈ [-zσ/√n, +zσ/√n])=1-α

It is clear that the z-interval represents error bounds and describes measurement uncertainty.

Hi,

I was going to ask a question very similar to your disscusion. Fortunately, i found it this first.

I have one week thinking and searching for literature in this topic and, surprisingly, this is scarce.

It is curious confidence interval consider only sampling error but not measurement error. Moreover, when a lot of instrument in my field (Psychology) have systematic measurement error. For now, I was working in how integrate in CI formula a correction with a measurement error.

Regards

Gabriel Alonso González-Medina

I assume that what you meant by “measurement error” is “calibration error” of your instruments. If this is the case, measurement error may be characterized by the Type B standard uncertainty (SU) based on manufacturer’s specifications, whereas sampling error may be characterized by the Type A SU. Then you can employ the law of propagation of uncertainty (LPU) to combine all SUs. The LPU is valid regardless of CIs or uncertainty theories.

The unification of frequentist and Bayesian inference is possible, at least in measurement uncertainty analysis, please see: "A new Bayesian method for measurement uncertainty analysis and the unification of frequentist and Bayesian inference" available from: https://www.researchgate.net/publication/344552280_A_new_Bayesian_method_for_measurement_uncertainty_analysis_and_the_unification_of_frequentist_and_Bayesian_inference?channel=doi&linkId=5f7fd8a5458515b7cf71d5ec&showFulltext=true