To add to the previous answers, I can also recommend this summary of reporting guidelines:

https://easystats.github.io/bayestestR/articles/guidelines.html

(one can also find a gentle introduction to Bayesian methods with R on this website, useful for students and beginners :) with Daniel Lüdecke and Mattan S. Ben Shachar)

The short answer is Yes- the Bayes factor is really better than the p-value.

The main reason is that the P-value (classic statistical inferences) simply answers a wrong hypothetical question that we only use as an unfortunate substitute for the actual question of interest.

The actual question of interest is "Given the available data, how confident are we that the null hypothesis is true?", i.e. what is the probability (likelihood) of the null hypothesis being true given the observed data, i.e. P(B/A).

Classic statistics tries to answer a different question: "How probable would the observed or stronger data (test statistics) to occur if the trial repeated many times given the null hypothesis is true." ,i.e. what is the probability of the getting the observed data given the null hypothesis being true, i.e. P(A/B). It is assumed that if the null hypothesis is true, then the probability of getting the observed data is high (95%, or 99%, etc), and vice versa, if the null hypothesis is not true (wrong), then we would not likely observe the data because the probability of observing would be low (5%, 1%, etc).

In general, P(B/A) can differ from P(A/B) by any factor. Thus, classic statistic inference simply substitutes the main question of interest for another question, but assumes these questions are equivalent, P(B/A)=P(A/B). This is wrong.

The posted question was the subject of extensive discussion in the literature. It comes down to the validity of the current prevalent statistical hypothesis testing methodology,and it has deep methodological implications.

It is unlikely, though, that Bayesian approach will replace the prevalent frequentist one in the near future, mainly because of practical difficulties of deriving the prior probabilities before the test commencement.

Thanks a lot for your answer Alexander Kolker . So I'll read more about bayes factor. Do you think that there are other concrete methods of interest to replace the p-value?

Alexander Kolker,

« The actual question of interest is "Given the available data, how confident are we that the null hypothesis is true?" »

I don't think that is correct. This is already a misinterpretation. In most cases, the parameter space is continuous (or has at least on continuous dimension). In such (typical) cases, the probability of any point hypothesis is zero. It is usually nonsense to ask for the probability of a point-hypothesis, and this is also not what Bayesian analysis is done for. Instead, what we want is the posterior distribution - over the entire parameter space.

I further think that the wise choice a the (assumed!) null hypothesis in classical statistical tests helps to see if a relevant aspect of the data is interpretable (sufficiently confident). These are often extremely basic questions, that are only about the "direction" of an effect relative to the chosen null (eg. "the signal scatters around some unknown value; we have a single observation m (from a sample of size n) - is the data at hand sufficient to at least interpret the sign of m-H0 (or not even that)?" For instance, for H0=0 we would see if the data suffices to distinguish a positive from a negative signal).

Jochen Wilhelm

...and the confidence in the null hypothesis H being true or not is expressed through the conditional posterior probability P(H/A) given the event (new piece of information ) A or its distribution.

As you mentioned, indeed, for a continuous random variable the probability of its specific value is 0. But this fact has nothing to do with the Bayes posterior probability, and is irrelevant in this discussion.

In general, the conditional probability P(H/A) is NOT equal to the probability P(A/H).

This posterior inverse probability P(H/A) is not based on frequency. This is known as subjective probability (likelihood). It changes depending on the overall state of knowledge. At the same time, it is this probability (likelihood) that is usually the primary interest in hypothesis testing, but not how likely are the observed data given the hypothesis.

For the Fisherian frequency hypothesis testing, no matter how many data you have, the p-value reflects the frequency probability that you would expect to get that data set or worse with multiple repeats of the experiment. With the Bayesian approach, you are measuring the uncertainty about the hypothesis being true. So, the conclusions could eventually converge as experimental evidence accumulates for or against the hypothesis to be tested. However the interpretations and the meanings still remain very different.

Anyway, thank you for your contribution to the discussion.....

Dear Alexander Kolker ,

thank you for your comments. I still have some questions:

"P(H/A) is NOT equal to the probability P(A/H)" - that's absolutely clear. But:

"This posterior inverse probability P(H/A) is not based on frequency. This is known as subjective probability (likelihood)."

1) As I know it, is "likelihood" the word to emphasize that a probability statement is about a known fact (an actual observation). It is the "hypothetical probability (density)" for a random variable X (with known distribution), having the realization x, where x is the value of the actual observation. There is nothing particularily "subjective" with this. Given we consider the "known distribution" as a model for a frequency distribution, the likelihood is frequentist tool. It is used in huge class of likelihood-ratio tests. It is as "subjective" as the choice of the null that should be tested. I think this is what you mean with "At the same time, it is this probability (likelihood) that is usually the primary interest in hypothesis testing".

Even stronger, the likelihood is a "frequency accumulator" (follwoing from the central limit theorem, I think). By this property it converges to Dirac delta-function with the mass at that point that is the "true value" in the frequency interpretation of probability.

2) The inverse probability is not the same as the likelihood. It is the product of prior and likelihood, normalized to give unit integral.

Here you miss that this statement is strictly restricted and holds under H0 only:

"For the Fisherian frequency hypothesis testing, [...] the p-value reflects the frequency probability that you would expect to get that data set or worse with multiple repeats of the experiment"

That's utmost important. As the p-value is a realisation of a random variable (P; like x is the realization of X), the distribution of P is known only under H0 (it's uniform). Only under this assumption I can say that observing small p-values is unlikely (because of the uniform distribution we can say Pr(P

Jochen Wilhelm

I think our discussion deviates more and more from the original posted question- Is Bayes factor really better than p-value.

I was trying to answer this question the simplest way by highlighting that classic Fisherian statistical inference addresses the question on the probability of observing the data given the stated hypothesis. However the question of real interest is inverse: what is the confidence/uncertainty of the stated hypothesis being true given the observed data (posterior probability of hypothesis vs the prior probability of the hypothesis). The frequentist and the Bayes questions are not equivalent.

Now, strictly speaking the term "probability of hypothesis" should not be used at all because the term 'probability' applies only to random variables with realizations in stable and repeatable conditions (statistical stability is of utmost importance here). However, a hypothesis is not a random variable-therefore it is better to say the confidence that the hypothesis is true or the degree of our uncertainty (with 0 no confidence and 1 total confidence with anything in between).

I used the term 'likelihood' to distinguish from 'probability' but this caused some confusion. Formally 'likelihood' is usually defined as the conditional probability of a process outcome by knowing the probability of a single event. I believe it is better to say 'confidence in hypothesis' or 'uncertainty" instead of the posterior probability of the hypothesis-but this is a historically used term in the Bayes literature.

I am not sure what you find misleading in the sentence "With the Bayesian approach, you are measuring the uncertainty about the hypothesis being true."

I believe that your suggestion "With the Bayesian approach, you update a probability distribution over the domain of possible hypotheses" will not be much better mainly because the probability distribution of a hypothesis does not exist- again, a hypothesis is not a random variable. But you can state your confidence or uncertainty about something expressed as value between 0 and 1.

Thanks again for your time and comments.

We have recently submitted a paper on this subject, where a pre-print is available:

Preprint Indices of Effect Existence and Significance in the Bayesian Framework

To summarize: it is good practice to report one index for effect existence and one for effect significance, as these are complementary perspectives. There's a Bayesian equivalent to the p-value that we suggest reporting, along with either a Bayes factor or the ROPE. When using Bayes factors, we also suggst not comparing it against a Null Hypothesis, but again against a ROPE (Region of Practical Equivalence).

Maybe some of the co-authors can chime in into the discussion, if they like... (Dominique Makowski and Mattan S. Ben Shachar )

Thanks for your answer Daniel Lüdecke , I'll read your paper with interest.

I think this paper will also help you

https://doi.org/10.1016/j.tourman.2017.11.011

Thanks for sharing this paper. I'll read this!

The use of Bayes factors is a Bayesian alternative to classical hypothesis testing, its use in testing superiority of one model over another is well know, but I still have doubts in it being used as a replacement to P-value because most of the times our research is focussed on proving significance or otherwise.

> because most of the times our research is focussed on proving significance

And that's part of the problem. There is a vast literatur on why "absence of evidence" (no significant result) does not imply the evidence of absence (i.e. there still might be a true effect). As we suggest in our paper, inference should be based both on "significance" (also in the sense of practical or clinical significance, not statistical significance) and "effect existence" (like magnitude and sign of an effect).

I wouldn't say that BF solve all these issues, and they have some limitations / assumptions as well (like well defined priors). But you need to be aware of the limitations of those indices you use to support your "evidence", and you also should consider the uncertainty underlying all estimation.

Some interesting reading:

Regina Nuzzo. Scientific method: Statistical errors. P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. Nature(506);2014: 105-152

Amrhein, V., Greenland, S., & McShane, B. (2019). Scientists rise up against statistical significance. Nature, 567(7748), 305–307. https://doi.org/10.1038/d41586-019-00857-9

Gelman Andrew, Greenland Sander. Are confidence intervals better termed “uncertainty intervals”? BMJ 2019; 366 :l5381 https://www.bmj.com/content/366/bmj.l5381

Amrhein V, Gelman A, Greenland S, McShane BB. 2019. Abandoning statistical significance is both sensible and practical. PeerJ Preprints 7:e27657v1 https://doi.org/10.7287/peerj.preprints.27657v1

I think it should be emphasized that Bayes factor can show the evidence in favor of the null hypothesis, not only in favor of the alternative hypothesis. Using classical statistical tests (calculating p-value), you cannot accept the null hypothesis, you can only disprove it. If p-value is high, it is not the evidence in favor of the null hypothesis. If the Bayes factor (BF01) is high, you have an evidence in favor of the null hypothesis. In view of this, if you are interested in the statistical basis to conclude that there are no differences, the Bayes factor is preferable.

I am not taking a side here, I think both Bayesian and Fisherian statistics have pros and cons. But I wanted to make a comment about the original question by Clement. The critique of the over-reliance on p-value as the ultimate arbitrator of results significance is not necessarily a critique of Fisherian statistics. Therefore Bayesian approach is not necessarily the alternative to the problems with p values. There are many Fisherian statisticians that criticize the p-value centered culture by emphasizing attention to effect size, power and report issues (confidence intervals for example).

Assaf, it sounds strange that you entitle people "Fisherian" who say they give attention to effect size and power. The sample estimate is a sample statistic, and interpreting this (as single realization from a particular experiment) as "the effect size" or and estimate thereof is inherently Bayesian and not Fisherian. Without any additional knowledge (or belief) about the plausibility of the estimate, it is impossible to judge if this is "good" or "bad" or how good/bad it is. It remains a sample statistic, and it may be as bad as one can imagine. And "power" in not a concept Fisher used or thought about. This concept requires to have an idea about a "true" effect (and actaully also a loss function) - and Fisher was aware that a scientist (at least a biologist or geneticist as Fisher) cannot be asked for. The "power" concept and error rates and all that is a construct of Neyman and Pearson, who Fisher must have seen as his philosopical opponents (I have not seen that Fisher and Neyman ever agreed on anything).

Depends on the situation

Dear Clément, I think this paper is possibly relevant, written by my great colleagues and mathematicians Jorge Tendeiro and Henk Kiers,

https://doi.apa.org/doiLanding?doi=10.1037%2Fmet0000221

A review of issues about null hypothesis Bayesian testing : Null hypothesis significance testing (NHST) has been under scrutiny for decades. The literature shows overwhelming evidence of a large range of problems affecting NHST. One of the proposed alternatives to NHST is using Bayes factors instead of p-values. Here we denote the method of using Bayes factors to test point null models as “null hypothesis Bayesian testing” (NHBT). In this paper we offer a wide overview of potential issues (limitations or sources of misinterpretation) with NHBT which is currently missing in the literature. We illustrate many of the shortcomings of NHBT by means of reproducible examples. The paper concludes with a discussion of NHBT in particular and testing in general. In particular, we argue that posterior model probabilities should be given more emphasis than Bayes factors, since only the former provide direct answers to the most common research questions under consideration. Best, Laura

Interesting paper, Laura Bringmann – and it's on RG here: Article A Review of Issues About Null Hypothesis Bayesian Testing

To add to the previous answers, I can also recommend this summary of reporting guidelines:

https://easystats.github.io/bayestestR/articles/guidelines.html

(one can also find a gentle introduction to Bayesian methods with R on this website, useful for students and beginners :) with Daniel Lüdecke and Mattan S. Ben Shachar)

Article The Bayesian Methodology of Sir Harold Jeffreys as a Practic...

Article Semantic and cognitive tools to aid statistical science: Rep...

Article Best uses of p-values and complementary measures in medical ...

Article Bayesian Inference: Understanding Experimental Data With Inf...

Article Détente: A Practical Understanding of P‐values and Bayesian ...

https://www.nature.com/articles/s41562-019-0807-z

https://academic.oup.com/psychsocgerontology/article-abstract/75/1/45/5033832

New paper on this topic in Nature Neuroscience:

Article Using Bayes factor hypothesis testing in neuroscience to est...

The essential part of the Bayesian analysis is the prior. The question remains open what prior to use. If I an convinced that there cannot be any effect, I would use a prior of the form Pr(effect=0) = 1 - and I would not even need any data to conclude that there is no effect (I was convinced of that before anway). If I am not that sure that no effect exists, I will get a posterior that spreads over some range of possible effect sizes (again an informative prior can considerably impact how close this range will be to "no effect") and one needs some assumption of a "minimaly relevant effect size" (in addition to a reasonable and defendable prior).

If the prior has little or no weight (e.g. it's not very informative and the sample size is relatively large), there is not much numerical difference between confidence intervals and credibility intervals (a major difference is in multiplicity correction). The result is an interval (the narrower the interval, the more information does the observed data provide about the effect size), and it must be judged if this interval extends into ranges that are considered "relevant". I don't see how the fact that the interval is a credible interval rather than a confidence interval is simplifying or even circumventing this judgement. It's usually much simpler to calculate a confidence interval, so what is the benefit of the extra work (setting up and defend a reasonable prior, calculationg MCMC to find the posterior)?