Can you suggest a continuous probability distribution that has non-finite entropy measure?
More specifically, suppose the random variable X has probability density function f(x) defined on the real-line R; i.e., f(x)>0 on R and integral of f(x) on R is unity. Provide an example of a density function f(x) (or may be a family of distributions), for which E[|log f(X)|] is not finite, where E[|log f(X)|] is the expected value of the random variable |log f(X)|; i.e., E[|log f(X)|] = integral of f(x)|log f(x)| on R.
Wouldn't a Cauchy fit this criteria? Since the integral in E[|log f(X)|] = ∫f(x)|log f(x)|dx diverges due to the crazy tails from the pdfs
Sujit K. Ghosh Sorry, it was a Friday, I was thinking about the moments hahahh, you're absolutely right
There is an answer at https://math.stackexchange.com/questions/4017267/does-any-probability-distribution-have-an-entropy-defined
that apparently provides an example for a discrete distribution. One would guess that this can be manipulated to provide a continuous distribution with the required property.
Hi David: good to hear from you and thanks for the citation to math.stackexchange site. Yes, we can modify the discrete distribution to create such an example.
More, generally, if we create a family of such continuous distributions, it would serve as test cases for Bayesian and Frequentist methods where typically the consistency (of posterior or MLE) hinges on the assumption of finite KLDs.
For example, we could consider log-pareto type distribution for a random variable X such that log(X) has a Pareto distribution, for which I think it will have infinite entropy for any positive scale parameter and the shape parameter in the range (0, 1).
You may need to look at the logic of that situation. It looks as if you would have a situation where maximum likelihood applied to data might not work, but where maximum likelihood applied to a transformed version of the data would work according to the usual theory. But that can't happen. A guess is that the usual theory could be extended to cover the case of having a transformation that gives a distribution with finite entropy.
Perhaps, we may consider a pdf like
(4/3 /sqrt(pi))*e^(-1/x^2) / x^6, for any real x (well-defined for x=0).
Then its entropy is infinity as
E[|log f(X)|] = ∫f(x)|log f(x)|dx diverges.
Though, this is quite a make-up (contrived ) example.
And, perhaps, this can be parameterized by replacing x with x-mu
https://www.wolframalpha.com/input?i=integrate+%284%2F3+%2Fsqrt%28pi%29%29*e%5E%28-1%2F%28x-7%29%5E2%29+%2F+%28x-7%29%5E6+from+x%3D-inf+to+%2Binf
https://www.wolframalpha.com/input?i=integrate+x*%284%2F3+%2Fsqrt%28pi%29%29*e%5E%28-1%2F%28x-7%29%5E2%29+%2F+%28x-7%29%5E6+from+x%3D-inf+to+%2Binf
and, the power of '6' in the denominator, x^6, might be some even power of x^k, k = 4, 6, 8, 10 etc. with an appropriate normalizing constant depending on 'k'.
Sorry, actually the suggested pdf refers to a random variable that is the square of the inverse gamma distribution, so of course its entropy is finite.
- Badges
- Science topic
- Similar topics
- Chemistry, Organic
- Organic