Hello,
Is this fair to report 95% confidence intervals in frequentist approach when we have doubts about representativeness of the sample, etc?
Using other words... having just one analysis how can we know that our results are within the 95% range of what is going on for the whole population?
What is your opinion on that?
Well, in fact I don't see that a CI offers so much as many people claim it would do. It does give some additional information to what a p-value alone gives (what is again actually very little and often not even a very interesting information!), but it's not a game-changer. It provides a range of hypotheses (assumed values of a coefficient within some statistical model) under which the observed data isn't considered "too unlikely" (or the other way around: for hypotheses outside of the CI, the observed data is considered "too ulikely"). Depending on the design of the experiment, the fromulation of the statistical model, and the sample size, this gives some hint if the amount of data observed is sufficient to rule out some possibly relevant hypotheses.
With increasing sample size, CI and credible interval approach each other, so especially when the sample size is rather large (and a reasonable prior is not too exotic), it is ok to interpret a CI as an approximate credible interval. Note that, however, the credible interval still does not make a claim about a "true population parameter" but instead expresses our (reasonable) believes about the value of a coefficient in a model we use to describe some aspect of reality.
The confidence interval (the particular interval you calculated from your particular sample) makes no claim about what's going on in the population. Very similar to a p-value, it makes a claim about the data, given an assumption of how the population looks like. It's not the fault of the CI to be interpreted wrongly.
In a Bayesian analysis, you can incorporate doubts about the representativeness when deriving the credability interval. However, this may not be possible in any case, and if there are serious doubts, I wouldn't iterpret too much in the results anyway.
Thanks a lot Jochen,
Could you please explain how you interpret CI for the specific data? What information does it convey? Or is just something to be reported but does not offer much in terms of explanation or interpretation of the results?
Regards
Witold
Well, in fact I don't see that a CI offers so much as many people claim it would do. It does give some additional information to what a p-value alone gives (what is again actually very little and often not even a very interesting information!), but it's not a game-changer. It provides a range of hypotheses (assumed values of a coefficient within some statistical model) under which the observed data isn't considered "too unlikely" (or the other way around: for hypotheses outside of the CI, the observed data is considered "too ulikely"). Depending on the design of the experiment, the fromulation of the statistical model, and the sample size, this gives some hint if the amount of data observed is sufficient to rule out some possibly relevant hypotheses.
With increasing sample size, CI and credible interval approach each other, so especially when the sample size is rather large (and a reasonable prior is not too exotic), it is ok to interpret a CI as an approximate credible interval. Note that, however, the credible interval still does not make a claim about a "true population parameter" but instead expresses our (reasonable) believes about the value of a coefficient in a model we use to describe some aspect of reality.
I like CIs say in regression analysis as they give the degree of uncertainty in the effect size which I mainly interested in - p does not do do that.
In random effects analysis attention focuses on the variances (and covarainces) and not just the means - there can be large differences in the credible intervals and confidence intervals as the latter are often based on normality assumptions which may not be appropriate for variances (which cannot go negative) and correlations (which are bounded between 0 and 1) - the credible intervals can by asymmetric if estimated say as a Gamma distribution in MCMC.
Confidence intervals can be asymmetric,* too. You can, for instance, calculate a confidence interval for a variance (based on the chi²-distribution), or for a rate of a Poisson process (based on the Poisson-distribution), or for a proportion (based on the binomial distribution), all of which will be asymmetrical.** Just to name a few.
In random-effects models, confidence intervals are usually determined by bootstrapping, and this also usually results in asymmetric intervals for variances. I think, if you use some recent software that is able to fit a mixed- or random-effects model, then there is no risk to get confidence intervals for variances that are based on normality assumptions (I could be wrong here - I don't know all the sofware, for sure).
---
*asymmetric means here: the point-estimate is not in the centre of the interval. The interval itself is simply an interval, defined by an upper and a lower limit; there is actually nothing related to symmetry.
** except for a proportion of exactly 0.5, as nitpickers may notice.
Maximum Likelihood as a particular case of Bayesian inference (depending on priors and other assumptions) may produce CI with equal values than CrI. Still, not the same (=results does not imply =definitions). Check:
https://stats.stackexchange.com/questions/230810/when-does-the-maximum-likelihood-correspond-to-a-reference-prior
and similar resources.
Inference with non-probabilistic samples has been discussed, as sampling procedure is unknown. Alternatives have been explored. Check:
https://www.cbs.nl/-/media/imported/documents/2015/44/predictive-inference-for-non-probability-samples.pdf
Under certain conditions a 95% CI will more-or-less be identical to a Bayesian probability interval with a particular prior or set of priors (e.g., flat/uninformative priors ). Also as you acquire more data the Bayesian and frequentist interval will tend to be similar (almost regardless of priors). However, there are situations where this will break down, and generally one doesn't want to use completely uninformative priors (if one is doing Bayesian analysis).
There are also some frequentist analyses that approximate a more principled Bayesian analysis. For example adding 2 successes and fails to the data for a CI for a binomial proportion approximates a Bayesian analysis with a Beta(2,2) prior.
Thus I think the link between the two is dangerous unless you know what you are doing (in which case it may be more sensible just to compute the Bayesian interval).
"...is dangerous unless you know what you are doing..." - well said :) This applies to all parts of science, data analysis, and statistics...
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS