My practical experience suggests that that different priors do not usually make much difference unless there is very little information in the data. Indeed different results could be seen as a feature and not a problem so that you could do a sensitivity analysis with different priors.

There is a nice example of this in Clayton and Hills Statistical Methods in Epidemiology page 119. They look at the elevated risk of leukaemia from living close to the nuclear reprocessing plant at Sellafield. The find the most likely estimate of risk to be 16 and with 3 vague priors the 90% posterior probability ranges from 5.5 to 37. But as they argue no epidemiologist would argue that a risk of 2 and 1000 are equally probable. So they redid the analysis with three more realistic priors (eg 90% limts of 0.3- 2.0) and their 90% posterior probabilities range from 0.97 to 3.33. They conclude "the Bayesian approach has therefore shown that such a small study cannot lead to identical beliefs in the scientific community. The posteorir distrribution is too influenced by prior belief and too litle by data' The number ofobserved cases was 4 against an expected 0.25..

Hi dear,

To understand the role of Bayesianism in statistical inference and assess its ability to improve scientific reasoning, we need to understand and appreciate the principles, advantages and disadvantages of frequentist statistics school.

somtimes the frequentist schoolm and Bayesian give us the sime results, another point is: the statistics methods are just a tool used by praticien to validate the hypothesis studies.

A part of your question is under a big problem of: statistical significance vs. clinical significance.

Best Regards

Hello Chellai Fatih,

Certainly the distinctions between frequentist and Bayesian approaches are important, and I have a number of thoughts on that matter. However, the concern being addressed here was on the ability of Bayesian inference itself to provide a consistent result, as per the two papers cited.

The "brittleness" is caused by using a "wrong" model in the Bayesian analysis. I'm not sure if one should even consider having a "wrong" model. The analysis *assumes* that the model is correct, and noone ever said that every conclusion will be fine when a "wrong" model is assumed.(*) So the problem is extermal to the analysis and not a consequence of the analysis. The nice thing aout a Bayesian analysis is that even different more or less plausible models can be considered all together.

---

(*) Also a frequentits analysis is based on the assumption of some model.

And how do we know if we've used the correct model?

In practice: We don't know. A model is more or less plausible. We should use the most plausible model we can imagine. The invention of such a model and the rating of its plausibility is outside of any analysis, in which this model is used (as an assumption).

Or in different words: there is no "correct model" for real data in the real world. In the analysis, the assumed model is the correct model, per definition; so there is no "wrong model" in the analysis.

If we don't know if the model is correct, we don't know if we have a mismatch and thus that Bayesian inference will fail. If we knew that we had the right model, we wouldn't need to test it. And the results of this study reject the idea that there is no "wrong model." But let's see where this discussion leads, as others join in.

assalam alaikum dear

With regard to my experience in the field of experimental design I have already used the Bayesian methods in estimating the parameters,The strength of this method lies in its reliance on the past information and experiences about the parameters and combining it with the current information you will get good estimators , In addition these results also depends on the type of the prior p.d.f(informative ,non informative)and its distribution used in the analysis .Finally i think comparing this method with another methods of analysis you will find out which the best way for inference.

Hello Nuha abdulrazzaq Yahya,

Your response, while interesting, seems to be a little disconnected from the specific topic at hand. The issue is whether Bayesian inference itself is trustworthy. It seems to be. You said yourself that you have used it numerous times, as have so many others.

It seems to work, and yet mathematically, it isn't necessarily expected to. Minor changes in prior probability or model selection apparently can result in vastly different outcomes, to the point where two researchers doing almost identical research could end up reaching the opposite conclusion.

Thanks Mr. Daniel Goldman

From the outset, I said that as far as my research is concerned, it is also possible to obtain different results by changing the previous function, but what i mean is that you should compare with other methods of inference in order to ascertain the validity of Bayesian approach.

I think I understand what you are suggesting now. Thank you.

you well com sir

Dear Daniel,

I am still struggeling with your sentence " If we don't know if the model is correct, we don't know if we have a mismatch and thus that Bayesian inference will fail. ".

A model is a model. It is correct only in an abstract environment, fixing the assumptions underlying the model, in a math-world. There is no logical connection between the model and the world, so there is no possibility to attribute terms like "correct" or "wrong" to a model regarding anything real. A model can be very useful (or very misleading) to describe particular aspects of reality, in a particular range of applicability. The question "correct or not?" is illogical in this context. The question can only be "useful or not?" (or better: how useful? and in what circumstances?).

Deciding to use a Bayesian model allows us to state beliefs on something (the parameters in that model, that have some useful meaning to us) and transform or update these beliefs according to observations. I really don't see how one can attribute words like "correct" and "wrong" to beliefs. We can sometimes check if beliefs are more or less justified. They are justified, when additional experience does not considerably change them and when the behaviour based on these beliefs is sensible and leads to the expected results.

What is it that we're trying to do with Bayesian analysis? Model real world phenomena. Regardless, the study suggests that it's not just a model mismatch that's the problem. Small perturbations in either the model or the prior can result in chaotic behavior.

Yes; they suggest that we can just use a coarser model, but can we really?

The study by Owhadi shows that chaotic results can be obtained from numerical solutions of a complex problem, in particular, when the solution is highly sensitive to initial conditions. This is not new and is not caused by or a failure of Bayesian theory.

> This is not new and is not caused by or a failure of Bayesian theory.

While I agree that the cause is more fundamental, to say that it is not a failure of Bayesian inference doesn't seem quite right. This paper, and others, suggest that we have to be very careful about our model selection, or even small differences in prior probabilities can result in completely different outcomes of the process.

And yet... Bayesian inference seems to be successful in a great deal of cases, even when we're not careful. So that's odd.

No. It is not odd. Complex and ill-defined models are subject to instability. There is no magic in a Bayesian model. The magic is skill and comprehension.

Okay. Let's put it this way. How would you go about making sure that Bayesian inference has not given you the result that it has because of the prior probability selection?

Bayesian analysis fortunately depends on the professional quality. So, it may not strong all the time. If you do right it should be very strong. Recall that priors are a little part of the work. There is no correct likelihoods as the case of priors.

But if two different priors, even when arbitrarily close together, can result in vastly different posterior probabilities, how do you know you've come to the "right" conclusion? Should we just have faith?

Ok, back to the cited paper. As far as I understood do the authors claim that the brittleness occurs when moments are estimated that don't exist in the underlying data-generating process (last two paragraphs on page 569). As they also state elsewhere, almost all pracical problems are about the first two moments, the brittleness does not seem to be of any great practical relevance. Even for some prototype examples of distributions without moments like the Cauchy, the problem has little practical relevance because there we usually have to deal with a truncated form (or with censored data) - and with both limitations we have a "real" data-generating prcess that will have at least the first two moments.

I am happy to get corrected if I understood this wrongly or missed something important (what is not quite unlikely here).

Further, a good Bayesian analysis always includes a sensitivity analysis w.r.t. the prior. If the posterior changes significantly for slight changes in the prior, this is a sign that one should not trust the analysis. Such a finding tells us that we should rethink the model in the particular case. So in this case we don't know if we got the "right" conclusion (or if the analysis pushes our believes in the "right" direction, or if our model class is meaningful in the context of observable data), but we are warned not to put much confindence in any conclusion based on this analysis (using this model class).

> As far as I understood do the authors claim that the brittleness occurs when moments are estimated that don't exist in the underlying data-generating process (last two paragraphs on page 569).

I'll reread and try to double check.

> As they also state elsewhere, almost all pracical problems are about the first two moments, the brittleness does not seem to be of any great practical relevance.

This statement would make sense if we can make the model "coarse" enough. But can we really do this?

> Further, a good Bayesian analysis always includes a sensitivity analysis w.r.t. the prior.

Agreed, Jochen Wilhelm. But there's a problem. How many priors do you test? 10? 100? A million? Even a countably infinite number of tests would constitute a set of measure zero. Is a finite number of test points really enough, if even the tiniest of perturbations can have a significant impact on posteriors?

Maybe it is enough. But I'm not convinced. By the way, I may be coming off as too strong about this topic, but I absolutely do appreciate this discussion and the time that you have taken to provide your thoughts on the topic.

My practical experience suggests that that different priors do not usually make much difference unless there is very little information in the data. Indeed different results could be seen as a feature and not a problem so that you could do a sensitivity analysis with different priors.

There is a nice example of this in Clayton and Hills Statistical Methods in Epidemiology page 119. They look at the elevated risk of leukaemia from living close to the nuclear reprocessing plant at Sellafield. The find the most likely estimate of risk to be 16 and with 3 vague priors the 90% posterior probability ranges from 5.5 to 37. But as they argue no epidemiologist would argue that a risk of 2 and 1000 are equally probable. So they redid the analysis with three more realistic priors (eg 90% limts of 0.3- 2.0) and their 90% posterior probabilities range from 0.97 to 3.33. They conclude "the Bayesian approach has therefore shown that such a small study cannot lead to identical beliefs in the scientific community. The posteorir distrribution is too influenced by prior belief and too litle by data' The number ofobserved cases was 4 against an expected 0.25..

Kelvyn,

I do agree that in a lot of peoples' experience, prior probability selection does not seem to be a problem. But mathematical analysis of the process itself suggests that it should be. That's why I am wondering if we can resolve the apparent issue (that doesn't actually seem to show up too often) by reconstructing the process in a paraconsistent logic. Of course, doing so would require reconstructing the models being tested, and so on, in that logic, which is a whole other matter.

If θ = (θ1,... θi, ...θk)T is of the prior parameter vector, I would check θ and all θ(±i) where θ(±i)=(θ1,... θi±di, ...θk)T and di is a small change considered just relevant for θi. This would make a total of 1+2k estimates that have to be determined. If for all "minimally relevant" changes in alle prior parameters the result is within ±d, I would consider the estimate robust enough. One could extend this to all 1+(2k)! possibilities to add and subtract the d's.

I see your concern, that it might still be that there are other points inbetween for which the estimate might be "outlying" ("britteling away" I woud say). So It can still be that the estimate is problematic or untrustworthy but we are not warned by the sensitivity analysis. However, I find it difficult to imaging a case where you will see not much changes for small d's, but drastic changes for smaller d's. If the d's are not "small" you would anyway accept a huge uncertainty in the estimate, so it should not matter much what the actual value is.

Jochen Wilhelm, it does seem odd, but at the same time, it seems odd to me that selecting a measure zero set on which to test sensitivity is in any way sufficient.

Of course, let me ask you another question. Would you be interested in the results of trying to reconstruct Bayesian inference in a paraconsistent logic? It seems like a fun topic, regardless of its necessity. Perhaps that really should have been the focus of my question in the first place.

Daniel, you said,

"That's why I am wondering if we can resolve the apparent issue (that doesn't actually seem to show up too often) by reconstructing the process in a paraconsistent logic."

There is no resolution. the brittleness paper is a red herring dragged over Bayes' theory. It is applicable to any other kind of analysis where insufficient information, poor data, or ill-posed questions are concerned. If the calculation is chaotic in one analysis system it is chaotic in all others if restricted to the same data and conditions. You cannot invent a solution by expression of another logic or prior.

I'm sure you're right. But how do we make sure that our results are not the product of chaotic behavior? I'm still not convinced that a sensitivity analysis using a set of measure zero is enough.

how do we make sure that our results are not the product of chaotic behavior?

Skill and comprehension. If you know what you are doing and how to do it, there will be no unpleasant surprises. The best sensitivity analysis is being your own strongest critic.

Human perception and evaluation of data and evidence is horribly poor. That's one reason why we use rigorous statistical tests. You're basically saying that we can accept that our results aren't the product of chaotic behavior, because our gut instinct says so.

Daniel Goldman

I do not think that this (coming to a judgement) is just a technical issue that is resolvable by a single test. Rather it is about convincing theoretical and empirical argument in conditions of (often considerable ) uncertainty, and this is a social activity for a wider scientific community. And we must be willing to say when the evidence is not compelling. I know that this is taking us away from your immediate concerns , but it is the 'real' context in which we work. For what it is worth, I have set out my views on what is achievable with quantitative analysis in a messy world in

Chapter The practice of quantitative methods

But at the same time, it is important to do work to improve how Bayesian methods work theoretically and in practice and I wish you well in that quest.