According to Fisher [1], “… probability and likelihood are quantities of an entirely different nature.” Edwards [2] stated, “… this [likelihood] function in no sense gives rise to a statistical distribution.” According to Edwards [2], the likelihood function supplies a nature order of preference among the possibilities under consideration. Consequently, the mode of a likelihood function corresponds to the most preferred parameter value for a given dataset. Therefore, Edwards’ Method of Support or the method of maximum likelihood is a likelihood-based inference procedure that utilizes the mode only for point estimation of unknown parameters; it does not utilize the entire curve of likelihood functions [3]. In contrast, a probability-based inference, whether frequentist or Bayesian, requires using the entire curve of probability density functions for inference [3].
The Bayes Theorem in continuous form combines the likelihood function and the prior distribution (PDF) to form the posterior distribution (PDF). That is,
posterior PDF ~ likelihood function × prior PDF (1)
In the absence of prior information, a flat prior should be used according to Jaynes’ maximum entropy principle. Equation (1) reduces to:
posterior PDF = standardized likelihood function (2)
However, “… probability and likelihood are quantities of an entirely different nature [1]” and “… this [likelihood] function in no sense gives rise to a statistical distribution” [2]. Thus, Eq. (2) is invalid.
In fact, Eq. (1) is not the original Bayes Theorem in continuous form. It is called the "reformulated" Bayes Theorem by some authors in measurement science. According to Box and Tiao [4], the original Bayes Theorem in continuous form is merely a statement of conditional probability distribution, similar to the Bayes Theorem in discrete form. Furthermore, Eq. (1) violates “the principle of self-consistent operation” [3]. In my opinion, likelihood functions should not be mixed with probability density functions for statistical inference. A likelihood function is a distorted mirror of its probability density function counterpart; its usein Bayes Theorem may be the root cause of biased or incorrect inferences of the traditional Bayesian method [3]. I hope this discussion gets people thinking about this fundamental issue in Bayesian approaches.
References
[1] Edwards A W F 1992 Likelihood (expanded edition) Johns Hopkins University Press Baltimore
[2] Fisher R A 1921 On the ‘Probable Error’ of a coefficient of correlation deduced from a small sample Metron I part 4, 3-32
[3] Huang H 2022 A new modified Bayesian method for measurement uncertainty analysis and the unification of frequentist and Bayesian inference. Journal of Probability and Statistical Science, 20(1), 52-79. https://journals.uregina.ca/jpss/article/view/515
[4] Box G E P and Tiao G C 1992 Bayesian Inference in Statistical Analysis Wiley New York
There are too many problems with this claim to respond to. Where do I begin. Repectfully, you seem have cobbled together bits and pieces from AWF Edwards and RA Fisher to represent all of likelihood theory. The entire likelihood curve - not just the inflection point - describes the variability around the maximum value. One likelihood curve is meaningless for inference. It is the relative comparison of two likelihood functions, each calculated from a different model based on competing hypotheses. It is precisely the areas under the curves that provide the information needed to determine whether the two (say) hypotheses differ and by how much.
Patrice Showers Corneli
Thank you for your comments. Indeed, “The entire likelihood curve - not just the inflection point - describes the variability around the maximum value.” Without the likelihood curve, the mode (the maximum value) cannot be determined. However, as far as I understand, the likelihood-based inference uses the mode only. In other words, the principle of maximum likelihood only considers the maximum value. Furthermore, the principle of maximum likelihood or Edwards’ method of support operates entirely on likelihood functions. It does not involve probability density functions. In contrast, the original Bayes Theorem operates entirely on probabilities, and the original Bayes Theorem in continuous form is merely a statement of conditional probability distribution [4]. Equation (1): posterior PDF ~ likelihood function × prior PDF, is actually called “reformulated BayesTheorem” in measurement science. Box and Tiao [4] provided a proof for converting the original Bayes Theorem in continuous form into Equation (1). However, as I discussed in my paper [3], their derivation “is far from rigorous”. My main concern is thatEquation (1) is a mixture of a likelihood function and a probability density function, like a mixture of apples and oranges.
As I mentioned briefly in the introduction section of my paper [3], there has been a debate in measurement science regarding the use of Bayesian approaches for measurement uncertainty analysis. The Joint Committee for Guides in Metrology (JCGM) tried to revise the GUM based on Bayesian statistics without success. So, it is a big puzzle why Bayesian approaches are controversial. I suspect that mixing likelihood function and probability density function is a fundamental problem in Bayesian approaches. I hope this discussion can help resolve this puzzle.
Thanks Huang. I really appreciate your post and the opportunity to discuss stats theory and particularly likelihood. And I wonder if the lingo and short-hand formulas are confusing to researchers who are not trained in theory.
It is true that Maximum Likelihood Estimators point estimates (e.g the means of many common distributions including the normal) are a single point - the inflection. Other ways of estimating such model parameters also are point estimates. I very much like MLEs, however, because of their vary nice properties not shared by other methods: invariance of functions of the MLE, smallish variance, and especially I like the relatively robust behavior of MLEs in the face of model assumption violations.
More later on Bayes Theorem, ML inference and Bayesian Statistics.
I reread your post and understand it better now as I have been rereading some of what I once knew in an instant. I do think that this is a problem with technical terms. A probability density function (pdf) is not a probability. For each statistical distribution, the pdf is a function which when integrated over the interval (0,1) provides a probability of the data X given a parameter 𝜽.
In contrast, the likelihood is a function of the parameter 𝜽 given the data. The integration is over all parameter values given the data
By definition, If f(x/𝜽) is the joint pdf (or pmf for discrete data) of a sample X= (Xi,... ,Xn), then the likelihood function for observations X=x, is L(𝜽/x)=f(x/𝜽). (see for example, any textbook of theoretical statistical inference such as Casella and Berger).
So you are right that the Likelihood method does not calculate probabilities, per se, but the pdf, which is a function, not a probability, is a crucial part of the likelihood of a parameter (a hypothesis).
The difference is that the pdf treats the parameter or hypothesis as fixed and the data x as random variables. The likelihood function treats the data as fixed and the parameter to vary over all possible parameter values.
Similarly for inference, a single likelihood is a meaningless value but the ratio of the likelihood of two different hypotheses (the null and the alternative) is highly meaningful as we try to decide which hypothesis better fits the observed data.
Taking the difference in the natural logs of each hypothesis tells us which is preferred. A log likelihood difference of 2 means that one hypothesis fits the data more than 7 times better than the other. The meaning of the preference is clear.
Regarding bias in Bayes, it is my view - and that of other people in my main field (molecular phylogenetics), much Bayesian bias is the result of improper priors. The wrong prior guarantees bias in the posterior probability.
Your equation rightly shows that the posterior function is a function of the likelihood times the prior. And using prior knowledge seems intuitively sensible but it is not as easy to determine the proper function for the prior. Using a uniform distribution renders the posterior probability a likelihood.
I use both Likelihood and Bayes methods but for the wonderful quality of likelihood parameters (sufficiency, robust to model violation, low variance, invariance), I prefer likelihoods over probabilities.
Patrice Showers Corneli
Thanks for your input. I agree with you that MLEs have “vary nice properties”. MLE is compatible with the least-squares estimation in some (if not many) cases. I think MLE is the least controversial method in statistics. While null hypothesis testing and Bayesian approaches have been debated and challenged for years, MLE does not appear to be a subject of debate.
I also agree with you that “…Bayesian bias is the result of improper priors. The wrong prior guarantees bias in the posterior probability.” For example, the Jeffreys prior 1/sigma is improper prior. For the case where the given dataset comes from multiple observations and no prior information is available, the use of the Jeffreys prior 1/sigma results in the scaled and shifted t-distribution, which is a “biased” or “distorted” normal distribution. The validity of the Jeffreys priors has been a subject of debate even among Bayesians. D’Agostini (1998), a leading proponent of Bayesian methods in particle physics, argued, “…it is rarely the case that in physical situations the status of prior knowledge is equivalent to that expressed by the Jeffreys priors, …”. He further stated, “The default use of Jeffreys priors is clearly unjustified, especially in inferring the parameters of the normal distribution, ….”. However, if we use a flat prior (based on the principle of maximum entropy) instead of the Jeffreys prior 1/sigma in the reformulated Bayes Theorem, we will get an even more distorted posterior distribution than the scaled and shifted t-distribution. This puzzles me for a long time.
I also agree with you that “… a single likelihood is a meaningless value but the ratio of the likelihood of two different hypotheses (the null and the alternative) is highly meaningful …”. However, in the reformulated Bayes Theorem, a single (and the entire) likelihood curve is used to generate the posterior distribution, and no comparison of different hypotheses is made in this application.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression