In Bayesian statistics, power analysis is slightly different from frequentist power analysis. Specifically, when discussing Bayesian power in the context of correctly concluding in favor of the null hypothesis, we often refer to this process as Bayes Factor analysis or Posterior Probability analysis.

Bayesian power measures how well the data supports the null hypothesis compared to the alternative hypothesis, usually through the use of Bayes Factors or examining the posterior probabilities.

Key Terms and Concepts:

Bayesian Power:

Bayesian power in favor of the null hypothesis is typically conceptualized through evaluating the posterior probability of the null or through Bayes Factors. In practice, we summarize how the data supports the null hypothesis based on these measures.

Dear Jochen Wilhelm

I have conducted several Bayesian analyses on 2,000 simulated datasets, all generated as random samples from the same statistical population. In these simulations, the expectation is that the Bayesian indices I’ve chosen should, in principle, support the null hypothesis (H₀).

Naturally, not all analyses return results in favor of H₀. I am interested in quantifying the proportion of cases in which the analysis correctly indicates support for the null (e.g., when BF₁₀ < 1/3 under the true null).

What would be the appropriate term to describe this proportion?

Thank you in advance for your insight.

Jochen Wilhelm

Article A tutorial on Bayes Factor Design Analysis using an informed prior

this: https://osf.io/preprints/osf/d5kf3_v1

and this: https://doi.org/10.1080/00031305.2025.2467919

Jochen Wilhelm

One is not "free to choose a prior".

Prior probability distributions must be the least informative distributions. When executed properly, using least informative distributions eliminates your concern.

An objective Bayesian analysis requires thoughtfully selected priors and the chosen distribution(s) must be fully discussed and, or revealed.

These important questions deserve attention.

The aim of Bayesian probability theory is to compute parameter estimates for random variables in a model for the data and, or select a model(s) for data. These estimates describe states of knowledge - including the estimate uncertainties.

Question: Why must prior probability distributions be the least informative distributions?

This quote explains why prior probability distributions must be the least informative distributions.

“…the mathematical rules of probability theory are not merely rules for calculating frequencies of random variables; they are also the unique consistent rules for conducting inference (i.e., plausible reasoning)”

Probability Theory: The Logic of Science, E.T. Jaynes, G.L. Bretthorest ed, Cambridge University Press, 2003; p. xxii

More on uninformative prior distributions follows.

Question: It's not always clear what the least informative distribution is. This also often changes when the problem is parametrized in a different way. This can therefore not be a general rule. Am I wrong here?

When a problem is “parametized “in a different way”, one simply computes the least informative prior distributions that properly account for the new “way”.

The determination of the least informative prior distribution(s) is clear.

It is clear because each parameter in the model for the data requires a prior probability distribution. Someone understands the mathematical model. Someone designed and conducted the data collection. Someone understands potential noise sources in the measurement. Often credible parameter estimates exist from similar models and measurements. All of these are testable.

It is clear because Prior probability distributions represent our state of knowledge about a parameter. This includes our state of ignorance about the parameter.

Two useful ways to represent our state of ignorance are uniform prior distributions and uninformative prior distribution.

Laplace wrote, “When the probability for a simple event is unknown, we may suppose all values between zero and one as equally likely”. A uniform prior is a distribution where we can’t assign a difference in the possibility of any parameter value within the distribution.

Uninformative priors are defined as maximally uninformative prior distributions.

If the Central Limit Theorem applies to the problem, A Gaussian prior distribution is maximally uninformative.

E.T. Jaynes explained uninformative priors that conform with the principles of maximum entropy are useful. When computing uninformative prior distributions we take care to assign a prior probability that represents our state of ignorance.

“Whenever we assign uniform prior probabilities, we can say truthfully that we are applying maximum entropy whenever we assign a Gaussian sampling distribution, this is the same as applying maximum entropy for given first and second moments …whenever we assign a binomial sampling distribution, this is mathematically equivalent to assigning the uniform maximum entropy distribution on a deeper hypothesis space.”

Sir Harold Jeffreys proposed a powerful approach to uninformative priors based on Fisher information. Jeffreys suggests that the reason that we favor the a prior with the least amount of information is simply because it has a higher probability than a needlessly more complex prior.

Question: If "objective Bayesian analysis" means that any analysis will come to the same conclusion based on the same data, then using the same prior distribution is sufficient. This distribution does not need to be "least informative". What is your definition of an "objective Bayesian"?

This comment does not take into account it’s not enough “that any analysis will come to the same conclusion based on the same data”. The conclusion must be consistent with plausible reasoning. Ill-chosen prior distributions render “the same conclusion based on the same data” an insufficient description for objective Bayesian analysis.

I can’t to improve on this definition. An objective Bayesian analysis:

* “The prior probabilities represent our prior information, and are to be determined, not by introspection, but by logical analysis of that information.

* Since the final conclusions depend necessarily on both the prior information and the data, it follows that, in formulating a problem, one must specify the prior information to be used just as fully as one specifies the data.

* Our goal is that inferences are to be completely 'objective' in the sense that two persons with the same prior information must assign the same prior probabilities”

Probability Theory: The Logic of Science, E.T. Jaynes, G.L. Bretthorest ed, Cambridge University Press, 2003; pp. 372-373

It has everything to do with priors because ill-chosen priors tainted with subjective, untestable assumptions are not possible when one is comm to plausible reasoning.

According to Berger & Sellke (1987), they assert that the Bayesian power of the test, defined as the probability of accurately affirming the null hypothesis when it is true, was determined by assessing the ratio of simulated datasets where the Bayes factor surpassed the threshold for evidence supporting the null hypothesis.