We prefer to use conjugate prior for the Bayesian model. Using variational Bayesian inference, conjugate prior is also preferred. However, in the update procedure of Bayesian inference, it seems that the conjugate prior property does not matter that much. My questions is: what is the significance of the conjugate prior in variational Bayesian inference?
In the paper titled "Bayesian group sparse modeling and variational Bayesian inference", the non-conjugate prior also works.
With a conjugate prior, you know the functional form of the posterior will be the same as the prior. This works well with variational Bayes as you can easily obtain the entire posterior from just point parameter estimates. Without conjugacy, the posterior can end up as a non standard and analytically intractable function, so you cannot easily look at the entire density using variational Bayes.
Conjugate priors of certain distribution parameters may be obtained analytically. For example one can analytically derive that the conjugate prior distribution of the mean of a poisson distribution is gamma. The mean of the binomial distribution with constant number of Bernoulli trials is known to have a prior beta distribution. That of the mean of a normal distribution with known variance is also a normal distribution, and so on. This reduces the tedium of the process of Bayesian inference to a large extent. A non-conjugate prior may not compare favourably with the conjugate distribution in a particular case.
To clarify a bit further, if you were to use a MH MCMC algorithm, then conjugacy is not needed, but using a conjugate prior typically allows use of Gibbs sampling instead of MH. For variational Bayes with a conjugate prior, you can use a less computationally intensive approach to obtain estimates of the posterior parameters (i.e. use a simpler optimization algorithm), which then gives you the entire posterior (assuming your conjugate likelihood and prior provides a reasonable model representation of the underlying data generating process) since you know the functional form of the posterior.
- Badges
- Science topic
- Similar topics
- Engineering
- Modeling