Thanks for your quick answer, Theo.

The first part of your answer - infering on the parameter eta in calculating p(eta | data) - is not what I was looking for. This is, indeed, described in many books.

I am interested in the marginal likelihood p(data) or L(data) for a normal distribution with unknown parameters mu and sigma2 (or precision tau). Starting with the data modeled as normal distributed the probability of data is p(data | mu, sigma2). When we assign a normal distribution on mu the probability is p(data, mu | sigma2, mu_0, sigma2_0), where mu0 and sigma0 are hyperparameters of the normal prior of mu. Again assigning an inverse-gamma prior on sigma2 we have p(data, mu, sigma2 | mu_0, sigma2_0, alpha0, beta0), where alpha0 and beta0 are the hyperparameter of the inverse-gamma prior.

Now I want to integrate out mu and sigma2. This would yield a marginal likelihood function of the data. The reason why I want a likelihood in this form is now to put this in a standard bayesian framework as likelihood function:

p(theta | data) = p(data | theta) x p(theta). Where p(data | theta) is the previously derived likeihood and theta are the parameter, which are NOT mu and sigma2.

Such a two-step process is common in changpoint models. (See Fearnhead, P., & Liu, Z. (2007). Online Inference for Multiple Changepoint Problems. Journal of the Royal Statistical Society, 69(4), 589–605.) They use a likelihood starting with a linear regression model with uncertain parameters. I want to use a gaussian model instead.

Thanks,

Sven.

Hi Sven,

Unfortunately I don't think its possible to get an analytical solution to the integral you want (although I would be very happy to be proved wrong)!

We can easily integrate over a normal prior on mu to obtain a normal marginal likelihood in terms of sigma2, mu_0 and sigma2_0. However, we cannot then integrate over an inverse gamma prior on the variance analytically, as the variance of the new distribution is a blend of sigma2 and sigma2_0. A way around this is to change your priors so that sigma2_0 is equal to z*sigma2 where z is a simple factor - in other words the variance of the prior is linked to the variance of the data. In this case when we integrate over a normal prior on mu we obtain a normal distribution with variance that is a simple function of sigma2 alone, meaning we can integrate over an inverse gamma prior on sigma2 no problem.

Another way of thinking about this is that you have a joint normal-inverse-gamma prior on mu and sigma2. If I remember correctly there is a derivation along these lines in Gelman, Bayesian Data Analysis. If not I can send you the details, but that won't really help in terms of a citeable document!

Bob

Hey Bob,

thanks for your insightful answer! Im not quite sure if there isn't a way to integrate over over an inverse-gamma prior on the variance. In my understanding Xuan (2007), p. 10-12 (attached) does exactly that.

But nevertheless I found a good description of the derivation of the marginal likelihood of a linear regression model with a normal-inverse-gamma prior on the beta and the variance in Greenberg (2014) (File "Question4.5" states the questrion and file "04Answer" the derivation is given).

If I am not completely wrong, one can replace the likelihood of the linear regression model by a gaussian likelihood and proceed in the same way described there to get the marginal likelhood of a gaussian with unknown mu and variance, right?

Thanks in advance!

Hey,

so as far as I can tell the derivation in Xuan agrees with my point - that you can't integrate over an inverse gamma prior on the variance unless the prior on sigma2 is linked to the prior on the mean (beta in this case). The horizontal arrow in Figure 2.1 of Xuan is crucial! It is this assumption that makes 2.22 still come out as Gaussian with variance proportional to sigma2, and not some horrible combination of parameters.

The fact that the priors are linked is very important, and often under-appreciated. To understand this, imagine a slider that allows you to change the value of sigma2 in the Xuan model. As you move this slider you change the variance of your likelihood AND the variance of your prior on beta. This is not the same as a model with independent priors on beta and sigma2, in which moving the slider would have no effect on the prior on beta.

Connecting the priors up in this way is exactly equivalent to assuming a joint normal-inverse-gamma prior on both parameters - in other words the results of Xuan and Greenberg that you found are identical (notice that Xuan 2.25 is the same as Greenberg's final answer, aside from a change of variables).

As for what you want to do, I am slightly confused. You want to assume a Gaussian likelihood and integrate over a normal-inverse-gamma prior on means and variances? If so then isn't the result in Greenberg already exactly what you want?

I'm interested to hear where this goes!

Bob

Hey Bob,

thanks for stressing out this point. It was not clear to me, so far. And I get that this is important to notice!

I totally agree that the results of Xuan and Greenberg are identical. However, both assume the data to be modeled by a linear regression. Hence, the likelihood is dependent on the regression parameter(s) beta. I want to assume my data to be i.i.d. Thus, I want to model my data by a normal distribution. Following Greenberg (2014) the integrable function would look like this:

f(y | mu, sigma2) x pi(mu, sigma2),

integrated over the paramters mu and sigma2.

In general, the result should be of the same form as described in Greenberg (2014) for the linear regression model. But some parameters will be calculated somewhat different. I finally found a paper where this is derived for a normal distribution - where the variance is replaced by the precision - with a normal-gamma distribution prior on the parameters mu and lambda (see attachment).

Thanks for your helpful answers, Bob!