.

a book :

Christian Robert

The Bayesian Choice: from Decision-Theoretic Motivations to Computational Implementation (2001) Springer-Verlag, New York

a complete set of slides for a course based on that book

http://www.ceremade.dauphine.fr/%7Exian/coursBC.pdf

.

The Bayesian Choice is a good book.

If you are searching for a machine learning oriented book, I would recommend

Christopher M. Bishop

Pattern Recognition and Machine Learning

http://research.microsoft.com/en-us/um/people/cmbishop/prml/

If you would like to explore the Bayesian nonparametric world, I recommend

Carl Edward Rasmussen and Christopher K. I. Williams

Gaussian Processes for Machine Learning

http://www.gaussianprocess.org/gpml/

And a tutorial on the Dirichlet process:

Yee Whye Teh,

Dirichlet Process

http://www.stats.ox.ac.uk/~teh/research/npbayes/Teh2010a.pdf

Have a look at Probabilistic Programming & Bayesian Methods for Hackers: http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/

Also "Bayesian Reasoning and Machine Learning" from David Barber is freely available online: http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage

.

spent some time fiddling with "Probabilistic Programming & Bayesian Methods for Hackers" as recommended above by Matti

quite interesting way of introducing the notions through "hands on" programming and definitely a useful complementary pedagogical approach to the more formalism-oriented approaches such as C. Robert's book

.