Assume you have two probability distributions X and Y.

X is the probability distribution of the phenomenon you mentioned. Y is an approximation / model of X.

The Kullback-Leibler divergence or relative entropy between X and Y, D_{KL}(X|Y), gives you the information loss when Y is used to approximate X.

If X is an information source with entropy H(X) and you use Y to model the source (e.g. because you do not have X), D_{KL}(X|Y) gives you the additional number of bits you will need to describe X. Hence you would need H(X) + D_{KL}(X|Y) bits

You can read more about the Kullback Leibler divergence in :

Cover, Thomas M., and Joy A. Thomas. Elements of information theory. John Wiley & Sons, 2012

If you questions is how much information Y "contains" about X you can look at the mutual information I(X,Y) which is the relative entropy between the joint distribution p(X,Y) and the product of the distributions p(X) p(Y).

I hope this answers your question.

Thanks for the reference. After posting I thought of cross entropy too, but now a new question arrises about how to get the probability density function of a machine learning model? Because the model usually only gives you the expected value.

Hi,

One relatively new approach is to use a codelength function as the measure of the information content of the data and the model. The Minimum Description Length principle then lets you select, objectively, the model which minimizes information loss while avoiding arbitrary model complexity. Peter Grunwald's book is a good, comprehensive introduction to MDL which lays out the links with standard methods for model evaluation and also classical information theory:

http://homepages.cwi.nl/~pdg/book/book.html

Neil,

VERY nice book: readable, friendly, very pedagogic; there are few books like this. I will certainly read it.

Thanks a lot!