Using R's arms package, I've run two Bayesian analyses, one with "power" as a continuous predictor (the 'null' model) and one with power + condition + condition x power. The WAIC for the two models are nearly identical: -.017 difference. This suggests that there are no condition differences.
But, when I examine the credibility intervals of the condition main effect and the interaction, neither one includes zero: [-0.11, -0.03 ] and [0.05, 0.19]. Further complicating matters, when I use the "hypothesis" command in brms to test if each is zero, the evidence ratios (BFs) are .265 and .798 (indicating evidence in favor of the null, right?) but the test tells me that the expected value of zero is outside the range. I don't understand!
I have the same models tested on a different data set with a different condition manipulation, and again the WAICs are very similar, the CIs don't include zero, but now the evidence ratios are 4.38 and 4.84.
I am very confused. The WAICs for both models indicate no effect of condition but the CIs don't include zero. Furthermore, the BFs indicate a result consistent with (WAIC) no effect in the first experiment but not for the second experiment.
My guess is that this has something to do with my specification of the prior, but I would have thought that all three metrics would be affected similarly by my specification of the prior. Any ideas?
I think I'm getting closer to answering my own question. It appears to lie in Chapter 12 of Kruschke's book. I'm going to keep digging.
The answer is rooted in the fact that the evidence ratio aka BF is dependent on the prior much more heavily than the CI is. If the prior is too broad, any particular value including zero is very unlikely, so analysis can produce a posterior in which that value is more likely even if it's at the edge of the CI.
As Andrew Gelman says on the BDA book, model selection still is a on going subject of research, and a problem not solved yet. I would trust more on the posterior intervals, given that they were grounded on probability axioms.
If the answer is useful, please recommend for others,
I just discovered that my issue has a name: Bartlett's paradox. Nice to see that I wasn't the only one confused by this state of affairs. It was first discussed in Bartlett (1957).
- Badges
- Science topic
- Similar topics
- Mathematics