Let X1,...,Xn iid random variables with distribution F(p), where p is some parameter. Due to some reasons, p is not observable directly (in the sense that there is no way to confirm whether p is static or dynamic). The challenge is to estimate the quantile of p to a given level, without assuming a particular distribution of p.
It seems that bootstrap is the only choice. So under the assumption that X1,...,Xn are not time-specific, bootstrap might not be bad choice. However, the estimate depends heavily on the number of bootstrap-draws. This results in unreliable quantile estimate.
However, assuming that p can be estimated by maximum likelihood method as well as any conditions necessary to ensure a consistent ML estimate, then we know that the ML estimate follows a normal distribution.
So is it appropriate to use the quantile of the ML estimate (using normal distribution) to estimate the quantile of p?
@ Christopher Imanto ,
p is a parameter. It does not have any distribution.
The MLE of p is a random variable and it has a distribution.
Note that the MLE of p is a function of the random variables
X1, X2 , ............... , Xn .
In the asymptotic distribution of sqrt(n) (p_MLE - p) , p is not a variable . MLE of p is the only variable.
Thus, p can not have quantiles. But MLE of p can have quantiles.
From the asymptotic distribution, you will obtain the quantile of MLE of p as a function of p & I(p).
Still you can not determine the value of the quantile of MLE of p since p is unknown.
The only approximation you can do is
" Substitute the value of the MLE of p, obtained from the observations, in place of p in the formula of the quantile of MLE of p."
I think the best way is to deal with a bayesian estimation; put your prior distribution of "p" and distibution of Xi, i=1,..,n. Because under this approach any parameter is considered as a random variable; (even with the classical inference "p" is always dynamic not static in the sense of variations).
The last sentence is not clear (at least for me)
The process of MLE assumes a distribution. For xbar for example the choice is normal.. Thus consistency of p by the same reasoning depends on the chosen distribution. This is not to say there is anything wrong with bootstrap. But your reasoning about consistency of p would seem to be circular. Best, D. Booth
In this problem, p is the parameter of the parent distribution, described by the probability function F(p), from which the n random observations
X1, X2 , ............... , Xn
have been drawn.
These random observations are regarded as random variables which are assumed to be iid.
Thus, each of them follows the probability distribution F(p) with parameter p.
Now, the ML Estimator of p is also a random variable.
But this random variable may not follow the normal distribution usually.
It will follow normal distribution if
X1, X2 , ............... , Xn
are iid normal variables i.e. if the form of F(p) is normal.
The parameter of this distribution depends upon the form of F(p).
From the form of the distribution, the form of quantiles of the ML Estimator is to be derived. Only then the estimates of the quantiles can be determined.
Thank you all for your answers. I exploit the asymptotic normality of ML estimators, i.e.
sqrt(n) (p_MLE - p) \rightarrow N(0, I(p)^{-1})
where I(.) is the fisher information.
While this (asymptotic) distribution serves well for estimating the confidence interval for p_MLE, it is not trivial to assume that the quantile of the normal distribution can serve as the estimate of p-quantile. For example, to estimate the 95% highest value for p, I can choose the 95-th percentile of the normal distribution. Note that the distribution of p is regardless unknown, only the asymptotic distribution of p_MLE is known.
@ Christopher Imanto ,
p is a parameter. It does not have any distribution.
The MLE of p is a random variable and it has a distribution.
Note that the MLE of p is a function of the random variables
X1, X2 , ............... , Xn .
In the asymptotic distribution of sqrt(n) (p_MLE - p) , p is not a variable . MLE of p is the only variable.
Thus, p can not have quantiles. But MLE of p can have quantiles.
From the asymptotic distribution, you will obtain the quantile of MLE of p as a function of p & I(p).
Still you can not determine the value of the quantile of MLE of p since p is unknown.
The only approximation you can do is
" Substitute the value of the MLE of p, obtained from the observations, in place of p in the formula of the quantile of MLE of p."
In term of Statistical inference, there is a contradictory when you say : P not observable (static or dynamic) and estimation of quantile of P, so we have to consider P as a random variable, because under ML estimation method p is a constant not variable.
As I commented above, you can deal with a bayesian inference approach, or like you noted to make a simulation study by which you can derive the quantiles of such parameter; because it's no acceptable to say the quantiles of unknown constant .
Best Wishes
Dhritikesh Chakrabarty . Exactly, that is what I proposed to do. From theoretical perspective, you are correct by addressing p as a parameter cannot have a quantile (or at least the quantile is p itself). From practical view, there is almost nothing in the real world, which is static. Almost everything has to some degree a deviation.
As you suggested, I derived the confidence interval to a given quantile level by calculating the fisher information. This is a formula with parameter p. I then replaced p with p_MLE. This is a valid estimator for the quantile of p_MLE, but may not be valid for the quantile of P (assuming P as a random variable).
Chellai Fatih . I'm not used to Bayesian approach, which is probably my primary reason not to use it. However, I understand the point why it might be useful in this case. The issue is that nothing is known about P and any assumption on the prior distribution is considered taboo. To make the matters worse, data is scarce as well and papers studying P contradict each other. Although, I'm not optimistic towards MLE either, this was the popular method measuring P (thereby ensuring comparability to other results). Which is why the question was formulated explicitly for MLE.
You can deal with confidence Interval, ( you 'll find in attached the detailled procedure to reach the end of estimation) ... But till now I'm not sure about the validity of such method given the conditions put on the parameter "p".
Could you give a reference to "asymptotic normality". (sum(x-xbar)^2)/n is an MLE of popn var. But I don't believe that it is asymptotically normal. Is there a proof?
@ David Eugene Booth ,
IF
X1, X2 , ............... , Xn
are n iid random variables following normal distribution
then sum(x-xbar)^2 follows a chi-square.sigma^2 distribution with n - 1 degrees of freedom where sigma^2 is the population variance.
i.e. {sum(x-xbar)^2/sigma^2} follows a chi-square distribution with n - 1 degrees of freedom.
However, if the parent distribution of
X1, X2 , ............... , Xn
is not normal then it is not correct to say that
{sum(x-xbar)^2/sigma^2} follows a chi-square distribution with n - 1 degrees of freedom.
However, if n is very large, the parent distribution of
X1, X2 , ............... , Xn
can be approximated by a normal distribution.
In that case, the distribution of {sum(x-xbar)^2/sigma^2} can be approximated by a chi-square distribution with n - 1 degrees of freedom.
Since chi-square distribution approaches normal distribution as n becomes larger
therefore for large n, the distribution of {sum(x-xbar)^2/sigma^2} can be approximated by a normal distribution.
@ Chellai Fatih
Parameter of a distribution is a constant for a specific situation.
If the situation changes but the type of distribution remains the same then the value of the parameter changes.
Thus, a parameter of a distribution can be regarded as situation dependent variable.
Note that If sample size changes, the value of the parameter also changes.
Thus, a parameter can be regarded / interpreted as a situation dependent variable.
Dhritikesh Chakrabarty Please cite some of these "theorems" in, say Bickel and Doksum. Thank you
David Eugene Booth At least I found a lecture note from MIT with proof of asymptotic normality for MLE. I just assume that theorems from MIT lecture notes are true and didn't bother to check the proof.
https://ocw.mit.edu/courses/mathematics/18-443-statistics-for-applications-fall-2006/lecture-notes/lecture3.pdf
@ David Eugene Booth ,
Christopher Imanto has provided the link https://ocw.mit.edu/courses/mathematics/18-443-statistics-for-applications-fall-2006/lecture-notes/lecture3.pdf
Which may be helpful to you.
Moreover, you can go through the book
"Advanced Theory of Statistics"
authored by Kendall and Stuart.
Dhritikesh Chakrabarty
The result was it converges in distribution. So exactly how do your claims follow?
As I said a reference from Bickel and Doksum or equivalent would suffice. Kendall and Stuart never use convergence in distribution. Bickel et al(1998) may suffice. D. Booth BTW I believe you said "approximated"
- Badges
- Science topic