The short answer is Yes- the Bayes factor is really better than the p-value.

The main reason is that the P-value (classic statistical inferences) simply answers a wrong hypothetical question that we only use as an unfortunate substitute for the actual question of interest.

The actual question of interest is "Given the available data, how confident are we that the null hypothesis is true?", i.e. what is the probability (likelihood) of the null hypothesis being true given the observed data, i.e. P(B/A).

Classic statistics tries to answer a different question: "How probable would the observed or stronger data (test statistics) to occur if the trial repeated many times given the null hypothesis is true." ,i.e. what is the probability of the getting the observed data given the null hypothesis being true, i.e. P(A/B). It is assumed that if the null hypothesis is true, then the probability of getting the observed data is high (95%, or 99%, etc), and vice versa, if the null hypothesis is not true (wrong), then we would not likely observe the data because the probability of observing would be low (5%, 1%, etc).

In general, P(B/A) can differ from P(A/B) by any factor. Thus, classic statistic inference simply substitutes the main question of interest for another question, but assumes these questions are equivalent, P(B/A)=P(A/B). This is wrong.

The posted question was the subject of extensive discussion in the literature. It comes down to the validity of the current prevalent statistical hypothesis testing methodology,and it has deep methodological implications.

It is unlikely, though, that Bayesian approach will replace the prevalent frequentist one in the near future, mainly because of practical difficulties of deriving the prior probabilities before the test commencement.

Thanks a lot for your answer Alexander Kolker . So I'll read more about bayes factor. Do you think that there are other concrete methods of interest to replace the p-value?

Alexander Kolker,

« The actual question of interest is "Given the available data, how confident are we that the null hypothesis is true?" »

I don't think that is correct. This is already a misinterpretation. In most cases, the parameter space is continuous (or has at least on continuous dimension). In such (typical) cases, the probability of any point hypothesis is zero. It is usually nonsense to ask for the probability of a point-hypothesis, and this is also not what Bayesian analysis is done for. Instead, what we want is the posterior distribution - over the entire parameter space.

I further think that the wise choice a the (assumed!) null hypothesis in classical statistical tests helps to see if a relevant aspect of the data is interpretable (sufficiently confident). These are often extremely basic questions, that are only about the "direction" of an effect relative to the chosen null (eg. "the signal scatters around some unknown value; we have a single observation m (from a sample of size n) - is the data at hand sufficient to at least interpret the sign of m-H0 (or not even that)?" For instance, for H0=0 we would see if the data suffices to distinguish a positive from a negative signal).

Jochen Wilhelm

...and the confidence in the null hypothesis H being true or not is expressed through the conditional posterior probability P(H/A) given the event (new piece of information ) A or its distribution.

As you mentioned, indeed, for a continuous random variable the probability of its specific value is 0. But this fact has nothing to do with the Bayes posterior probability, and is irrelevant in this discussion.

In general, the conditional probability P(H/A) is NOT equal to the probability P(A/H).

This posterior inverse probability P(H/A) is not based on frequency. This is known as subjective probability (likelihood). It changes depending on the overall state of knowledge. At the same time, it is this probability (likelihood) that is usually the primary interest in hypothesis testing, but not how likely are the observed data given the hypothesis.

For the Fisherian frequency hypothesis testing, no matter how many data you have, the p-value reflects the frequency probability that you would expect to get that data set or worse with multiple repeats of the experiment. With the Bayesian approach, you are measuring the uncertainty about the hypothesis being true. So, the conclusions could eventually converge as experimental evidence accumulates for or against the hypothesis to be tested. However the interpretations and the meanings still remain very different.

Anyway, thank you for your contribution to the discussion.....

Dear Alexander Kolker ,

thank you for your comments. I still have some questions:

"P(H/A) is NOT equal to the probability P(A/H)" - that's absolutely clear. But:

"This posterior inverse probability P(H/A) is not based on frequency. This is known as subjective probability (likelihood)."

1) As I know it, is "likelihood" the word to emphasize that a probability statement is about a known fact (an actual observation). It is the "hypothetical probability (density)" for a random variable X (with known distribution), having the realization x, where x is the value of the actual observation. There is nothing particularily "subjective" with this. Given we consider the "known distribution" as a model for a frequency distribution, the likelihood is frequentist tool. It is used in huge class of likelihood-ratio tests. It is as "subjective" as the choice of the null that should be tested. I think this is what you mean with "At the same time, it is this probability (likelihood) that is usually the primary interest in hypothesis testing".

Even stronger, the likelihood is a "frequency accumulator" (follwoing from the central limit theorem, I think). By this property it converges to Dirac delta-function with the mass at that point that is the "true value" in the frequency interpretation of probability.

2) The inverse probability is not the same as the likelihood. It is the product of prior and likelihood, normalized to give unit integral.

Here you miss that this statement is strictly restricted and holds under H0 only:

"For the Fisherian frequency hypothesis testing, [...] the p-value reflects the frequency probability that you would expect to get that data set or worse with multiple repeats of the experiment"

That's utmost important. As the p-value is a realisation of a random variable (P; like x is the realization of X), the distribution of P is known only under H0 (it's uniform). Only under this assumption I can say that observing small p-values is unlikely (because of the uniform distribution we can say Pr(P

Jochen Wilhelm

I think our discussion deviates more and more from the original posted question- Is Bayes factor really better than p-value.

I was trying to answer this question the simplest way by highlighting that classic Fisherian statistical inference addresses the question on the probability of observing the data given the stated hypothesis. However the question of real interest is inverse: what is the confidence/uncertainty of the stated hypothesis being true given the observed data (posterior probability of hypothesis vs the prior probability of the hypothesis). The frequentist and the Bayes questions are not equivalent.

Now, strictly speaking the term "probability of hypothesis" should not be used at all because the term 'probability' applies only to random variables with realizations in stable and repeatable conditions (statistical stability is of utmost importance here). However, a hypothesis is not a random variable-therefore it is better to say the confidence that the hypothesis is true or the degree of our uncertainty (with 0 no confidence and 1 total confidence with anything in between).

I used the term 'likelihood' to distinguish from 'probability' but this caused some confusion. Formally 'likelihood' is usually defined as the conditional probability of a process outcome by knowing the probability of a single event. I believe it is better to say 'confidence in hypothesis' or 'uncertainty" instead of the posterior probability of the hypothesis-but this is a historically used term in the Bayes literature.

I am not sure what you find misleading in the sentence "With the Bayesian approach, you are measuring the uncertainty about the hypothesis being true."

I believe that your suggestion "With the Bayesian approach, you update a probability distribution over the domain of possible hypotheses" will not be much better mainly because the probability distribution of a hypothesis does not exist- again, a hypothesis is not a random variable. But you can state your confidence or uncertainty about something expressed as value between 0 and 1.

Thanks again for your time and comments.

We have recently submitted a paper on this subject, where a pre-print is available:

Preprint Indices of Effect Existence and Significance in the Bayesian Framework

To summarize: it is good practice to report one index for effect existence and one for effect significance, as these are complementary perspectives. There's a Bayesian equivalent to the p-value that we suggest reporting, along with either a Bayes factor or the ROPE. When using Bayes factors, we also suggst not comparing it against a Null Hypothesis, but again against a ROPE (Region of Practical Equivalence).

Maybe some of the co-authors can chime in into the discussion, if they like... (Dominique Makowski and Mattan S. Ben Shachar )

Thanks for your answer Daniel Lüdecke , I'll read your paper with interest.

I think this paper will also help you

https://doi.org/10.1016/j.tourman.2017.11.011

Thanks for sharing this paper. I'll read this!

The use of Bayes factors is a Bayesian alternative to classical hypothesis testing, its use in testing superiority of one model over another is well know, but I still have doubts in it being used as a replacement to P-value because most of the times our research is focussed on proving significance or otherwise.

> because most of the times our research is focussed on proving significance

And that's part of the problem. There is a vast literatur on why "absence of evidence" (no significant result) does not imply the evidence of absence (i.e. there still might be a true effect). As we suggest in our paper, inference should be based both on "significance" (also in the sense of practical or clinical significance, not statistical significance) and "effect existence" (like magnitude and sign of an effect).

I wouldn't say that BF solve all these issues, and they have some limitations / assumptions as well (like well defined priors). But you need to be aware of the limitations of those indices you use to support your "evidence", and you also should consider the uncertainty underlying all estimation.

Some interesting reading:

Regina Nuzzo. Scientific method: Statistical errors. P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume. Nature(506);2014: 105-152

Amrhein, V., Greenland, S., & McShane, B. (2019). Scientists rise up against statistical significance. Nature, 567(7748), 305–307. https://doi.org/10.1038/d41586-019-00857-9

Gelman Andrew, Greenland Sander. Are confidence intervals better termed “uncertainty intervals”? BMJ 2019; 366 :l5381 https://www.bmj.com/content/366/bmj.l5381

Amrhein V, Gelman A, Greenland S, McShane BB. 2019. Abandoning statistical significance is both sensible and practical. PeerJ Preprints 7:e27657v1 https://doi.org/10.7287/peerj.preprints.27657v1