Dear all,
I am (re-)analyzing some previously published findings for a specific phenomenon (not really relevant here - this is a methodological question) in the area of JDM for a meta-analysis. Typically, researchers have used multiple regression or linear mixed effect models to indicate the influence of one predictor on the criteria. As I have read some papers about the advantages of using Bayesian stats and also meta-analysis, I thought I could give it a try and apply my limited but growing knowledge about Bayesian stats to this problem to assess how much evidence is actually backing the reported -and mostly significant- p-values.
However, I do not have the original raw data, but only the reported parameters in the literature (b, SE, p, t, beta etc.). The question is, whether I can compute a Bayes Factor (BF) to assess the strength of the influence of one specific variable. I know that typically one would test different regression models against each other and compute the relative change / improvement of BF. Sadly, I do not have this luxury, but can only use the available data.
Thus, I came up with the following idea: I could use the test-statistic of the parameters (t; giving evidence whether the effect is statistically significant from 0) and sample size to infer the BF for a specific parameter. However, I wonder if this approach is valid as I have never seen this in a published paper / blog and neither does the documentation of the R package BayesFactor (Morey & Rouder, 2014) mention such an application. Yet, there is the possibility of compute a BF based on a one-sample t-test against 0, which is equivalent to the t values of the parameter reported in the models.
Any Bayesian out there, who could help me out? Advice is much appreciated!
best,
Tom
Hi Thomas,
Interesting question!
First, it's worth noting that people rarely calculate exact Bayes factors. A Bayes factor is equal to the evidence of model 1 divided by the evidence of model 2, and it is rarely possible to calculate the exact model evidence in practice. Simple linear multivariate regressions are actually one of the few examples in which the true evidence (and hence the true Bayes factor) can be calculated exactly, as long as a simple form of the model is assumed (see Wikipedia on "Bayesian linear regression" for details). But for multilevel or mixed effects regressions you're out of luck. More complex methods for estimating the evidence require draws from the posterior distribution, and cannot really be done with just point estimates.
Instead of calculating exact Bayes factors, people tend to calculate approximate values of the evidence using simple heuristics. The most common of these is the Bayesian Information Criterion (BIC), which is an approximation to the model evidence under some simple assumptions about the prior. If you took the BIC of one model and compared with the BIC of another then you could obtain an estimate of the Bayes factor - just do BIC1 minus BIC2 to obtain an estimate of the log Bayes factor for model 1 over model 2. (A technical detail and added bonus is that the estimate of the Bayes factor is likely to be more accurate than the estimates of the individual model evidences used to produce it).
All you need in order to calculate the BIC is the number of observations, the number of free parameters, and the likelihood at the maximum likelihood. I'm assuming you know the first two, and you should be able to get hold of the maximum likelihood from the parameter estimates in the original paper along with the form of the likelihood (which I'm assuming they give somewhere). This may be a bit of work, but its the only way.
ps. I would stay away from ideas relating to test statistics, as this confuses a very Frequentist idea with the more Bayesian approach of calculating Bayes factors. The two things are philosophically very different.
Bob
Hi Thomas,
Interesting question!
First, it's worth noting that people rarely calculate exact Bayes factors. A Bayes factor is equal to the evidence of model 1 divided by the evidence of model 2, and it is rarely possible to calculate the exact model evidence in practice. Simple linear multivariate regressions are actually one of the few examples in which the true evidence (and hence the true Bayes factor) can be calculated exactly, as long as a simple form of the model is assumed (see Wikipedia on "Bayesian linear regression" for details). But for multilevel or mixed effects regressions you're out of luck. More complex methods for estimating the evidence require draws from the posterior distribution, and cannot really be done with just point estimates.
Instead of calculating exact Bayes factors, people tend to calculate approximate values of the evidence using simple heuristics. The most common of these is the Bayesian Information Criterion (BIC), which is an approximation to the model evidence under some simple assumptions about the prior. If you took the BIC of one model and compared with the BIC of another then you could obtain an estimate of the Bayes factor - just do BIC1 minus BIC2 to obtain an estimate of the log Bayes factor for model 1 over model 2. (A technical detail and added bonus is that the estimate of the Bayes factor is likely to be more accurate than the estimates of the individual model evidences used to produce it).
All you need in order to calculate the BIC is the number of observations, the number of free parameters, and the likelihood at the maximum likelihood. I'm assuming you know the first two, and you should be able to get hold of the maximum likelihood from the parameter estimates in the original paper along with the form of the likelihood (which I'm assuming they give somewhere). This may be a bit of work, but its the only way.
ps. I would stay away from ideas relating to test statistics, as this confuses a very Frequentist idea with the more Bayesian approach of calculating Bayes factors. The two things are philosophically very different.
Bob
Here's a nice review of the topic. I recommend checking out the section ' indicator variable selection'.
http://www.esajournals.org/doi/pdf/10.1890/14-0661.1
@Bob, thanks for your insightful comment.
The BIC is a great idea and heuristic - I love it. However, I wonder about your ps: I have read several papers and documentations of tools (e.g. BayesFactor package in R, online BF calculator: http://pcl.missouri.edu/bayesfactor) about transforming test statistics in BFs. You would argue that these are not valid approaches at all?
@Brian: also thank you for the link to this paper. If I read it correctly, the mentioned section is concerned with MCMC, which I would use if I had the complete dataset and could compute it on my own. However, I do not have the original data set, but only the reported (sometimes quite sloppy, frequentist) results. Till now I had the impression that MCMC was not really feasible without original data - did I miss something here?
Hey Thomas,
maybe this could be of help with the t-tes idea:
http://www.indiana.edu/~kruschke/BEST/BEST.pdf
Bayesian estimation supersedes the t test. John K. Kruschke, Journal of Experimental Psychology: General, 2013, v.142(2), pp.573-603. (doi: 10.1037/a0029146).
Hello Thomas,
I completely agree with Robert's answer. In your case, I think that using BIC would be the best course of action, as long as you are aware of its limitations and assumptions.
I attached an article that explains very clearly how to use BIC for model comparison. The part from page 796 is the one that most interests you.
Incidentally, this is one of the very best introductory articles to bayesian statistics I have ever read, so I suggest you to read it from the start!
Thomas,
Thanks for the link to those papers on obtaining test statistics from Bayes factors, again, very interesting! I think for the sake of being controversial and provocative I would have to say that I do disagree with those methods on purely philosophical grounds (which is not quite the same as arguing they are 'invalid').
In the pure Bayesian approach we start from the premise that probability can be used to express a degree of belief about a particular situation (e.g. a parameter value or hypothesis). Our prior beliefs are modified by the data to obtain an updated set of beliefs. When comparing between models, the posterior (equivalently the Bayes factor under a uniform prior) tells us which of these particular set of models we believe to be true. So, to be clear, we are always talking about the particular experiment at hand.
In contrast, in the pure frequentist viewpoint we cannot represent beliefs by probabilities. Parameter values and hypothesis are fixed and unknown, rather than being true variables, and it does not make sense to give a probability to a fixed value. All we can do is come up with a decision rule that - if followed dogmatically - controls the proportion of times that we come to an incorrect conclusion. Thus, we cannot state anything about the particular experiment at hand. We can only follow the rule and control our long term error rate.
Attempting to come up with the distribution of a Bayes factor as a test statistic seems like it is merging these two philosophies in a rather fuzzy and unsatisfying way. We first integrate over a prior, presumably meaning we are happy viewing unknowns as variables (the pure Bayesian view). Then we obtain a Bayes factor, which tells us everything we need to know about the problem at hand in the Bayesian viewpoint. But rather than sticking with the Bayes factor, we chicken out at the last second and decide to look at the distribution of this test statistic, moving to the worldview in which hypothesis are fixed but unknown. This seems like a method that is tempting to do from an implementation point of view, but actually is sat squarely on the fence from a philosophical point of view.
As I said, this is quite a purist critique of the method, and I'm sure some others will disagree!
Bob
- Badges
- Science topic