Dear Scott Lindauer,

I guess you can check the books by Robert Ash and Leon Brillouin. If I'm not mistaken both discuss the subject.

Regards,

Marcelo S. Alencar

Dear Scott Lindauer,

The Shannon entropy is not measure in that sense. The measure may be defined only on the algebra (that is on the set of elements with given of operations of addition and multiplication). The Shannon entropy is a measure of probabilistic uncertainty. This is a functional which satisfies appropriate axioms. See more detail for example J. Klir “Uncertainty and Information: foundations of Generalized Information Theory”, John Wiley & Sons Inc., 2006.

Si es una medida que se define considerando un sistema de eventos E= {e1, e2,…, ei} con una distribución de

probabilidades dada para cada evento del sistema, por ejemplo P(e) en cuyo caso se tiene:

H(E) = -∑ P(e) x log P(e)

El logaritmo es de base 2.

In the discrete framework there is no problem. For cotinuous variable it is not a measure. You need to divide by a u niform measure, associated to the a priori probability. However, see Cover and Thomas, Elements of Inf. Theory and Li & Vitanyi, An Introd. to Kolmogorov Complexity and its Applications.

A Mathematical Theory of Communication By C. E. SHANNON

The Bell System Technical Journal,Vol. 27, pp. 379–423, 623–656, July, October, 1948.

http://cm.bell-labs.com/cm/ms/what/shannonday/paper.html

6. CHOICE, UNCERTAINTY AND ENTROPY

page 10 may be what you want

Entropy H is a mathematical description of a information source. A measure based on entropy H is the Kullback-Leibler distance. http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence

I agree with Mostafa Monemizadeh: A measure based on entropy H is the Kullback-Leibler distance

There is a modified form of property (2) fulfilled by Entropy H, which is known as GROUPING AXIOM. see the books of information theory for more detail.

Sure: "The Shannon entropy is not measure in that sense" : That is : In the sense of the Mathematical Theory of Measure.

It is a "measure" (in the ordinary sense) of the uncertainty associated to a probability distribution. However, to define an entropy in a continuous framework, you need to introduce a measure, in the sense of the Mathematical Theory of Measure, in the underlying probability space. A probability is a function that assigns numbers to SETS; for that reason a probability is a measure.

If Scott is just thinking about the additive property of the Shannon entropy, there is an old paper of Renyi which may be helpful.

Your question is whether the entropy is a measure, i.e. a set function on a sigma-algebra Sigma verifying properties (1) and (2) above. (Incidentally (1) and (2) do not define a measure but only a pre-measure, to speak about a measure,

the property (2) must hold also for countably infinite disjoint unions).

The answer is definitely NO. And is no, regardless of X being a finite, a countably infinite or an uncountable set.

Therefore, you understand that your questions 1. and 2. cannot be asked since H is not a measure.

H is a function that is not defined on Sigma but on the set of all probability measures defined on Sigma. That is to say, for every probability measure mu on Sigma, H(mu) is a number quantifying the degree of indeterminacy conveyed by mu. It is in this loose sense that H is quoted as measuring something, not in the precise mathematical sense of a measure.

The best mathematically rigorous introduction to entropy I know (that remains accessible to the non-mathematician) is the marvellous small book of A. I. Khinchin, Mathematical foundations of information theory, Dover, New York (1957). (Mind that the entropy is a notion that was introduced in thermodynamics by Clausius and in statistical physics by Boltzmann in the first part of the 19th century and mathematically formalised by Kolmogorov in the 1930's. All the basic theory is thus already developed in Khinchin's book. More recent references but much more mathematically demanding are D. Duelle, Thermodynamic formalism, Cambridge University Press. If you are interested in the quantum counterpart, the basic reference is J. von Neumann, Mathematical foundations of quantum mechanics, Princeton University Press, or the book by Ohya and Petz, Quantum entropy and its use, Springer-Verlag.

Entropy is not measure ! Entropy is a function on probability space ! Actually it maps a pdf of a function to a real number. However a measure have to assign a set a real value. Hence this discussion has very simple answer. Entropy is not measure at all.

Shannon entropy is not a measure itself in the measure theoretic sense, rather it is used to measure probabilistic uncertainties.

Indeed, Shannon entropy is known as Shannon Diversity Index too. In the life sciences, this index is used to measure species diversity.

In continuation of discussion.

Entropy is not a measure (only of uncertainty). O.K. - But what may we say about mutual amount of information I(X,Y) = H(X)+H(Y) - H(X,Y)?

Nor Entropy or KL-Distance are measures in the measure theoretic sense.

Relative Entropy (aka Kullback-Leibler distance) does not give rise to a proper "Metric Space" since the triangle inequality - required for metric spaces - fails.

Cumulative pdf functions on an underlying space with defined ordering do form a useful metric space where "similarity" metric is the L norm .. maximum (supremum of) absolute difference -- that is Kolmogorov-Smirnov. The problem with Entropy is it gets very much more sensitive the closer you get.

I beg to differ

H=E(log(x))

The Lebesgue integral exists

A basic theorem in measure theory states that the Lebesgue integral of

A measurable function is measure

You confuse "measure" with "metric". Measure takes sets to R, metric takes pairs of points to R and must satisfy triangle inequality.

First of all correction E(log(p(x)) in my previous post.

James, one can induce a metric ftom measure. Think symmetric difference.

You appear to be referring to the space of x .. in that view changing from p to q is a change of measure .. expectation operator itself is changing .. entropy can be defined on distributions p (.) but in that sense it is a function and not a measure .. it is clear that relative entropy is not a metric on the hilbert space of distributins p (.) .. see above as Dimitiri explains this more clearly than I.