Very interesting topic. I agree with most of it, but also one has to take into account that Bayesian approaches:

1) Are based in likelihoods calculations, which is not something thought in school (i.e. is not trivial to learn)

2) gives you a population of answers, not a single one, which is both an advantage and a disadvantage.

3) Sometimes is over-parametrized, which for normal datasets is not a big deal, mainly with the computational power we have today. However, it is an issue, when dealing with massive datasets.

4) there is an important effect of the priors in the outcome. This is good if we are testing the hypothesis with different priors, but is a problem if we do not know much about the analysed data.

Cheers!

Thank you for this comment, I think is a very useful comparison between both methods, I would like to read this paper from William Bolstad (2007), if anybody share this paper with me, it will be a great help.

Very interesting topic. I agree with most of it, but also one has to take into account that Bayesian approaches:

1) Are based in likelihoods calculations, which is not something thought in school (i.e. is not trivial to learn)

2) gives you a population of answers, not a single one, which is both an advantage and a disadvantage.

3) Sometimes is over-parametrized, which for normal datasets is not a big deal, mainly with the computational power we have today. However, it is an issue, when dealing with massive datasets.

4) there is an important effect of the priors in the outcome. This is good if we are testing the hypothesis with different priors, but is a problem if we do not know much about the analysed data.

Cheers!

@ Osvaldo: It is not a paper is a book "introduction to Bayesian statistics", 2007.

Thank you Jose

Hi Jose

I think that computer power and learning is not (or will not be soon) a problem as computer power is growing almost every hour and peoples brain is also getting smarter in almost same rate:))

There was once a funny sentence in a paper from Rasmussen:

"the only difference between Bayesian and non-Bayesian methods is the prior, which is arbitrary anyway..."

Apart from that, a good paper on that topic would be:

"Objections to Bayesian statistics", Andrew Gelman, in Bayesian Analysis, 2008, Vol. 3, No. 3, pp. 445-450.

And also his book: "Bayesian Data Analysis".

This is clear that the quality of a Bayesian estimator can suffer from a poor prior, but this will be smoothed out by the number of samples. Alternatively, one can use a noninformative prior and just rely on the data. The advantage of a Bayesian approach is that we end up with a posterior distribution on the parameter to be estimated and a posterior predictive distribution.

@ Osvaldo : I find Bolstad's book to be one of the best around for teaching Bayesian statistics at an introductory level. He also shows the clear connections between Bayesian estimators (using specific priors) and Frequentist estimators that my students have found to be very useful.

Having a declared prior (as in Bayesian) is better than having an undeclared prior (as in Frequentist). I see only two limitations of Bayesian analysis:

1) the computation time is much longer - especially when data set gets larger

2) there is no user-friendly software (yes there are OpenBugs and others); a software that can be used by researchers who don't run run that particular software on a daily basis.

The attached paper is definitely a must read for those thinking about these issues.

"What is Bayesian Statistics and why everything else is wrong"..Michael Levine

https://www.math.umass.edu/~lavine/whatisbayes.pdf

.

two good reads :

David Lindley, The philosophy of statistics

Journal of the Royal Statistical Society. Series D (The Statistician), Vol. 49, No. 3(2000)

www.phil.vt.edu/dmayo/personal.../Lindley_Philosophy_of_Statistics.pdf

Andrew Gelman and Cosma Rohilla Shalizi, Philosophy and the practice of Bayesian statistics

British Journal of Mathematical and Statistical Psychology, 8--18(2013)

http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf

which insists on "model checking" as an essential (and often overlooked) step in statistical analysis

.

Though in general I'm a Bayesian fan (though I still think La Place deserves the naming rights, not Bayes), I would caution that Bayesian methods and the proliferation of Bayesian tools often tempt analysts to treat what are actually epistemic uncertainties as aleatoric. Some Bayesian advocates openly promote this approach with a uniform prior (e.g., treating ALL inputs to a simulation as uniform-distributed!). Properly, epistemic uncertainty analysis should not involve a probability distribution, regardless of the frequentist or Bayesian approach. The emerging literature on Uncertainty Quantification (applied to simulation) discusses alternative epistemic UQ methods, such as interval analysis, possibility theory, belief theory. These methods are certainly more straightforward for simulation, in which experiments are limited only by budgets, computer capacity and wall-clock time, than for real-world data analysis, where experimental data may be extremely limited.

Many people (including me) adopt the pragmatic approach of using a frequentist approach when sample size is sufficiently large, which in this case the effect of prior gets "swamped" by data , and adopt a Bayesian approach when sample size is modest/small but there are honest reasons to believe that the limited data can borrow strength from highly reliable indirect evidence (yes, it is subjective) expressed in the form of a suitable prior.

It's good to know some Bayesian statistics which sometimes comes in handy in applied work.

Do you want to predict or test hypothesis?

For prediction, I think the bayesian paradigm is hard to beat and most successful applications of Bayesian methods are in prediction.

For testing hypothesis, on the other side, there is still some room for discussion ;^)

I agree Juan, when it comes to predictions the Bayesian approach is preferable. That's because predictions involve integrating over the posterior of the model parameters. So instead of making predictions with the most probable parameter we take into account all possible values the parameter could take.

In principle a similar strategy is used to find hyperparameters. Hyperparameters control qualitative features of the data (such as the overall noise level for example). To construct the posterior distribution over hyperparameters we should integrate over the microscopic (or data generating parameters, which usually scale with the dimension of the data or sample size). This is often difficult in practice but in my experience can lead to a much more robust inference of hyperparameters.

By quantifying our uncertainty in terms of probability distributions we can perform these integrals. This offers a systematic way of inferring microscopic parameters, hyperparameters, and models.

I think the question Bayesian *versus* frequentist is wrong. We need both. We need to understand strengths and weaknesses of both. Bayesian statistics is very good for telling you what you should believe. It is "one person statistics". It is not so useful for telling other people what some data is telling us. It can also be very misleading when there are many parameters (or when parameters are infinite dimensional). Choice of prior is crucial and cannot be done by intuition. Mathematical study of *frequentist* properties of Bayesian methods is necessary in order to guide intuition. The problem is that Bayesian inference puts a lot of weight on what is actually rather soft information: prior knowledge about parameters, models. Remember that no models are true - but some can be useful, some are more useful than others. Life isn't easy. Whoever said it would be?

Richard I do agree!

Moreover, frequentist people claims the prior is arbitrary but what about the likelihood? When the weight of evidence provided by observations is not disputed, the likelihood function is widely considered to be the inter-subjective element of statistical inference. This is not the case in situations where the fundamentals of the science involved are disputable.

Yes, Patrizio, and I think this is what Fisher called the "type-II error".

I'd like to weaken two of Richard's statement:

" Bayesian statistics [...] is "one person statistics" "

Not neccesarily. If the prior is stated, the posterior tells everyone what one *should* (reasonably) believe based on *this* prior and *this* data. Also the prior can be inter-subjective or even "objective" with regard to the way it was derived from assumptions everyone would agree upon. A Bayesian reports what one should (reasonably!) believe, given clearly stated prior knowledge and the data. A frequentist can't tell you anything, except that you might keep an error-rate when rejecting null hypotheses (it is a purely decision-theoretic approach, kind of a long-run quality-assurance, not telling you anything directly about this individual result). When point estimates and confidence intervals are taken to make inference about the parameter under investigation, the interpretation is inherently Bayesian, silently assuming a flat prior.

"The problem is that Bayesian inference puts a lot of weight on what is actually rather soft information: prior knowledge about parameters, models."

This is only the case when little data is available. Larger amounts of data "override" the influence of the prior.

From Patrizio; Jochen and Fausto remarks it seems that none of the two discussed approaches is free from important error premises and prior problems. Should we try to innovate and propose other alternatives? Does it mean that many important researches and conclusions made in past may be seriously flawed if not manipulated in doubtful ways, as Fausto has warned us? ... This is important and it requires honest corrective actions. emilio

@ the question,"How [a] beginner can choose what to learn?"

First, learn logic. Logic teaches how to formulate premises and deductions. It also teaches induction or how to form the premises. Data analysis for purposes of answering a question requires unambiguous premises without hidden assumptions. Logic teaches how to avoid fallacies.

Statistical analysis has one of three purposes; cataloging, prediction, or control. It is necessary to know which purpose to form premises and design a study.

Second, learn probability. Data analysis shifts the logic statement from "If A then B" to "If A probably B." This change in statement means that a point value has little meaning, the distribution of B is all important. The distribution limits the ability to predict or control.

Third, Bayesian logic of probability will be your natural choice. Use frequentist methods when useful.