Hi.

I attach appendix B of "Bayesian Theory" by Bernardo & Smith, my preferred book on Bayesian statistics.  There you'll find a quite detailed comparison with other approaches to statistical inference.  I must admit that reading the entire book makes it easier to understand it, but I still believe that this appendix is one of the best comparisons.

Anyway, if you care about my personal view of the subject, please continue reading.  First, you must be explicit in formulating the question you want to answer.  Is it in the form P(data|model) or P(model|data)?  Note that if you happen to fall in the second category --- for example, you could ask yourself "what is the probability that the peak of my Gaussian model is between 1.5 and 2.5?" --- then the frequentist approach is excluded.  The reason is that statements in the form P(model|data) are forbidden (and ill-defined) in the frequentist approach, where only the data "fluctuate" while the parameters are not random variables, so that you cannot "attach" a probability to the model parameters.

The probability is operationally defined as the long-run relative frequency in this approach.  This definition is mathematically ill-defined, but works whenever it's applicable (i.e. when experiments can be repeated identically) thanks to the law of large numbers (the relative frequency tends in probability to the true probability), which holds whatever interpretation of probability you choose.  On the other hand, Bayesians define the probability as someone's degree of belief (which I admit it sounds bad, but it's not bad at all once you carefully think about it).  The law of large numbers ensures that the two definitions work equally well when the frequentist one is applicable, but the degree of belief is much wider.  For example, you can ask yourself what is the probability that tomorrow it rains if P = degree of belief, but not if P = frequency.

When dealing with statistical inference, the Bayesian paradigm is much better, but at the expense of computability.  The learning process is modeled by Bayes' theorem --- which gives the name also to the whole approach --- that states that the posterior probability of say your parameter µ given the data x and the rest of the info r is proportional to the likelihood function times the prior probability for µ:

p(µ|x,r) ~ p(x|µ,r) p(µ|r)

In multidimensional problems, if you want to get rid of the nuisance parameters (e.g. you have a Gaussian model but only care about µ and the variance is of no use) you have to integrate over them (using the corresponding priors as weight; in other words, priors define a metric for them, giving more weight to more likely regions). As multidimensional integration is non trivial, until few decades ago people preferred to use frequentist methods instead, as the latter pick the "best" value of each parameter instead of integrating, and reduce the problem to a minimization in multidimensional spaces, which computationally is _much_ easier.  However, modern numerical methods and computers are powerful enough that a full Bayesian treatment is affordable.  Not to mention that if you are lucky/smart enough, you'll have analytical solutions which save even more time than switching to minimization routines.

Coming back to inference, asymptotically the posterior peaks where the likelihood function peaks (so the max likelihood "principle" holds, but here it's a theorem).  But for very small samples the importance of the prior is not to be underestimated.  Usually, small samples are where the fun and the interest is, so this is an important point.  In addition, both semantically and numerically, P(rain|cloud) != P(cloud|rain) :-) so that the posterior is a different beast from the likelihood.  This means that, even when the max likelihood solution numerically coincides with the posterior peak, they have different meaning.  [If you've read about confidence vs credible intervals, you should understand what I mean.]

Finally, when it comes to decision theory, in particular hypothesis testing, the difference is huge.  Frequentist hypothesis testing is --- actually are, as there is more that one version of it --- so bad, that people are complaining about it since the very beginning (not to mention that different "bigs" never agreed with each other).  Despite from that, it is being taught nowadays... something that I hardly understand.  Recently the Science journal banned frequentist p-values (http://www.sciencebasedmedicine.org/psychology-journal-bans-significance-testing/) and I hope this will spread rapidly, as p-values are almost invariably misinterpreted and (even worse) misused by practically everybody (even experts from time to time).

[The most frequent misuse is when people compare two hypotheses, say H0 and H1, and after getting a small p-value for H0 alone (the "null hypothesis") they conclude that H0 is false and H1 is true.  The decision is not supported by the p-value, as the latter provides not enough information (in particular, it does not deal with H1 at all).  There is vast literature about pitfalls of p-values and hypothesis testing.]

Best regards,

Diego

Hey,

I suggest a simple example. 

Assume you have the follwing simple model:

y = x + n 

where the problem is the following: "Estimate x through observing different values of y which is a noisy version of x due to the random parameter n".

The Bayesian approach would tell you to see x as a random variable.

The frequentist (or deterministic) approach would tell you to see x as a deterministic unknown.

Bovens, L., & Hartmann, S. (2003). Bayesian Epistemology. Oxford University Press.

Dale, A. I. (1991). A History of Inverse Probability: From Thomas Bayes to Karl Pearson (Sources and Studies in the History of Mathematics and Physical Sciences). Springer.

Jeffreys, H. (1961). Theory of Probability. Oxford University Press.

Jeffrey, R. (2004). Subjective probability: The real thing. Cambridge University Press.

Samaniego, F. J. (2010). A Comparison of the Bayesian and Frequentist Approaches to Estimation (Springer Series in Statistics). Springer.

Stauffer, H. B. (2008). Contemporary Bayesian and Frequentist Statistical Research Methods for Natural Resource Scientists. Wiley.

(the last two are bolded just because they are the most directly relevant to your question.)

Given the potential confusion the first response might have as well as a possible need for expediency, I've attached a few sources so that you need not wait to buy or obtain as a library loan several books.

A discussion about the difference between the Bayesian and frequentist approach can turn rapidly to a kind of metaphysical discussion. I prefer the point of view of a user and I do not see a large difference for the most application without actual prior-information. I use the example of answer #1

y=x+n

I do not care about if x is determined or a random variable. It has a fixed value for the concrete situation.

The limit of distinction between randomness and determination is well illustrated by a pseudo random number being generated by numerical generator. The number is fully determined. But we can handle the number as random variable if we do not know the start value of the (pseudo) random generator.

As an user of both approaches, I think they are complementary from all points of view (epistemological, methodological, applied, educational ...), because together they provide you with a comprehensive view of the implications and possibilities of statistical inference; however, the Bayesian approach is more intuitive and easier to understand, while the frequentist approach is easier to implement. I think in the next blog you can find an excellent discussion of the topic, with very simple examples: http://stats.stackexchange.com/questions/2272/whats-the-difference-between-a-confidence-interval-and-a-credible-interval

Thank you all. I believe I will gain some insight now

Applicability of an abstract theory into the physical reality is in itself a philosophical issue. The variations of a mathematical formalism is not philosophical; it is just a methodological manipulation of axioms with the (objective) goal of representing reality adequately. In the case of probability interpretations, all of them are scientific and formal. The Kolmogorovian account has the status of ortodoxy due to its integration into measure theory and - as its founder argues, the real random processes can only be adequately idealized through infinite spaces; however, all the other interpretations of probability preserve a "measure" character of the probability as "measure of possibility"; among them, the frequentist one (including Bayesian) plays a major role in applications in sciences where statistical inference is used. I think the distinction between the two interpretations (with the implicit philosophical issues) is clearly caught here: http://plato.stanford.edu/entries/probability-interpret/ 

Hi.

I attach appendix B of "Bayesian Theory" by Bernardo & Smith, my preferred book on Bayesian statistics.  There you'll find a quite detailed comparison with other approaches to statistical inference.  I must admit that reading the entire book makes it easier to understand it, but I still believe that this appendix is one of the best comparisons.

Anyway, if you care about my personal view of the subject, please continue reading.  First, you must be explicit in formulating the question you want to answer.  Is it in the form P(data|model) or P(model|data)?  Note that if you happen to fall in the second category --- for example, you could ask yourself "what is the probability that the peak of my Gaussian model is between 1.5 and 2.5?" --- then the frequentist approach is excluded.  The reason is that statements in the form P(model|data) are forbidden (and ill-defined) in the frequentist approach, where only the data "fluctuate" while the parameters are not random variables, so that you cannot "attach" a probability to the model parameters.

The probability is operationally defined as the long-run relative frequency in this approach.  This definition is mathematically ill-defined, but works whenever it's applicable (i.e. when experiments can be repeated identically) thanks to the law of large numbers (the relative frequency tends in probability to the true probability), which holds whatever interpretation of probability you choose.  On the other hand, Bayesians define the probability as someone's degree of belief (which I admit it sounds bad, but it's not bad at all once you carefully think about it).  The law of large numbers ensures that the two definitions work equally well when the frequentist one is applicable, but the degree of belief is much wider.  For example, you can ask yourself what is the probability that tomorrow it rains if P = degree of belief, but not if P = frequency.

When dealing with statistical inference, the Bayesian paradigm is much better, but at the expense of computability.  The learning process is modeled by Bayes' theorem --- which gives the name also to the whole approach --- that states that the posterior probability of say your parameter µ given the data x and the rest of the info r is proportional to the likelihood function times the prior probability for µ:

p(µ|x,r) ~ p(x|µ,r) p(µ|r)

In multidimensional problems, if you want to get rid of the nuisance parameters (e.g. you have a Gaussian model but only care about µ and the variance is of no use) you have to integrate over them (using the corresponding priors as weight; in other words, priors define a metric for them, giving more weight to more likely regions). As multidimensional integration is non trivial, until few decades ago people preferred to use frequentist methods instead, as the latter pick the "best" value of each parameter instead of integrating, and reduce the problem to a minimization in multidimensional spaces, which computationally is _much_ easier.  However, modern numerical methods and computers are powerful enough that a full Bayesian treatment is affordable.  Not to mention that if you are lucky/smart enough, you'll have analytical solutions which save even more time than switching to minimization routines.

Coming back to inference, asymptotically the posterior peaks where the likelihood function peaks (so the max likelihood "principle" holds, but here it's a theorem).  But for very small samples the importance of the prior is not to be underestimated.  Usually, small samples are where the fun and the interest is, so this is an important point.  In addition, both semantically and numerically, P(rain|cloud) != P(cloud|rain) :-) so that the posterior is a different beast from the likelihood.  This means that, even when the max likelihood solution numerically coincides with the posterior peak, they have different meaning.  [If you've read about confidence vs credible intervals, you should understand what I mean.]

Finally, when it comes to decision theory, in particular hypothesis testing, the difference is huge.  Frequentist hypothesis testing is --- actually are, as there is more that one version of it --- so bad, that people are complaining about it since the very beginning (not to mention that different "bigs" never agreed with each other).  Despite from that, it is being taught nowadays... something that I hardly understand.  Recently the Science journal banned frequentist p-values (http://www.sciencebasedmedicine.org/psychology-journal-bans-significance-testing/) and I hope this will spread rapidly, as p-values are almost invariably misinterpreted and (even worse) misused by practically everybody (even experts from time to time).

[The most frequent misuse is when people compare two hypotheses, say H0 and H1, and after getting a small p-value for H0 alone (the "null hypothesis") they conclude that H0 is false and H1 is true.  The decision is not supported by the p-value, as the latter provides not enough information (in particular, it does not deal with H1 at all).  There is vast literature about pitfalls of p-values and hypothesis testing.]

Best regards,

Diego

I like  Bernardo and Smith. Christian Robert has discussion in his book, The Bayesaian Choice. From the other side of the street, D.R. Cox offers accessible discussioon of this and much more in his Principle of Statistical Inference. I'm going to assume you are interested in applied statistical issues in seismology. In that case you might also enjoy the accumulated wisdom in Cox and Donnelly's Principles of Applied Statistics. 

Many of the answers above focus on statistical inference rather than probability itself. For discussion of the latter, perhaps the best reference is the book of essays

Studies in subjective probability.by Henry Ely Kyburg; Howard E Smokler; New York; Wiley (1964),

especially that of de Finetti. Unfortunately this book is out of print, but may be available in libraries.

Going back to inference, a classic paper comparing the practical import of the Bayesian and frequentist approaches is

http://drsmorey.org/bibtex/upload/Edwards:etal:1963.pdf

I wrote an introduction explaining the importance of this paper:

http://link.springer.com/chapter/10.1007%2F978-1-4612-0919-5_33

Unfortunately I don't have an electronic copy to post.

@ Andrew Messing : Thanks for the papers.  I agree that they are very good to read.  Could you please provide full bibliographic references for the two chapters?

@ William Dumouchel : Thanks for the paper.

To me, a Bayesian approach is all about using a prior probability distribution, i.e. guide your estimate, based on your observations, using a prior distribution with a specified mean and (co)variance. If the variance is large, then the impact of your prior mean on your estimate will be small. 

I think Bain & Engelhardt (in Introduction to Probability and Mathematical Statistics) sum up the Bayesian position quite nicely when they say that the use of a prior "constitutes an additional assumption that may be helpful or harmful depending on its correctness".

I'm not sure about Frequentism, but I guess it would be more data driven, with no prior. 

http://www.stat.ufl.edu/archived/casella/Talks/BayesRefresher.pdf