Hello,
I ran GLMMs and Multilevel Bayesian model on the accuracy of participants. The p-value result (barely) significant (.048), while the Bayesian model does not include that factor as relevant (I used the "maximal" technique for mixed models by Barr et al, 2013).
I suggested that Bayesian tests are generally more conservative compared to p-values and require more evidence (Wetzels et al., 2011).
What are other possible explanations?
Thank you!
I disaggree with Ette Etuk, whether a p-value is below the criterion is a binary decision, significant or not.
However, I think the most important source of discrepancy is that Bayesian approaches use priors.
The p-value signifies that your data provides enough information to statistically distinguish the estimate from the hypothesized value (if this is zero it means that you can interpret the sign of the estimate, that is to infer the sign of the (unknown) true parameter value). This is a very different statement or conclusion than that from the Bayesian analysis that tells you that the (unknown) true effects is unlikely to be relevant (give a particular prior!).
That's because you are comparing oranges to apples
I'm pretty sure that if you compute a pvalue using the posterior the two will be pretty close (so you also avoid dichotomies)
Francesco Ruotolo , I need to point out a small but very relevant misconception in your answer: You write that "The frequentist approach (i.e. the p value) tells you what is the probability associated with the presence of an effect" -- this is wrong. It is THE KEY of the frequentistic appraoch that NO assumptions are made (or need to be made) about the probability of an effect. It is all and only about the data. The p-value is the probability of "more extreme data" (more specifically: of a statistic calculated from the data) under a specified model. This is the key difference to the Bayesian analysis that in fact assigns a probability distribution over the possible values of the effect size, and then uses the data to refine this distribution.
In neither case the question is whether or not an effect exists (except for the uncommon case of a Bayesian analysis putting a finite probability point mass at a zero effect). The frequentist will get an estimate (s statistic, a "data thing") and compare this to a hypothesized value (typically zero) to see if the data provide enough information to believe that the estimate is on the correct side of the hypothesized value. The Bayesian calculates the posterior distribution that represents her current beliefs about the effect size.
Besides the different assumptions underlying bayesian and frequentist approaches (as Jochen pointed out), I would like to add that there is not THE bayesian method to asses evidence for (or against) your hypothesis! Depending on your choice of method, the bayesian and frequentist results may differ more or less (especially if evidence is not very strong or ambigious). For example, the baysian statistic that is most closely related to the p-value is the "probability of direction", which tells you how much probability mass lies on one side of your hypothesized value (e.g. zero). The p-direction is linearly associated with the p-value. Other approaches like p-MAP, ROPE, Bayes Factor or combinations thereof are not and therefore may differ more or less from the p-value approach. Plese the the following simulation study on this topic, which illustrates the relationships.
Makowski, D., Ben-Shachar, M. S., Chen, S. A., & Lüdecke, D. (2019). Indices of effect existence and significance in the Bayesian framework. Frontiers in psychology, 10, 2767.
@Rainer doesn't he mean "evidence of absence " instead of the other way around?
Stefano Nembrini I did not refer to this point specifically, only that the interpretation of the bayesian results depend on the chosen method and may differ more or less from the frequentist approach (ceteris paribus).
But you make a valid point: the OP said that the frequentist approach showed evidence for the effect, but the bayesian evidence of absence. This would be odd under noncommital prior conditions. Maybe Tommaso Currò can clarify what he meant and show the results (maybe he just meant that the bayesian analysis did not show evidence for the effect and it is undecided [and not evidence of absence]).
- Badges
- Science topic
- Similar topics
- Social Psychology
- Attitude