I have a matrix 'A' with dimensions MXN and I want to calculate the pairwise Symmetrical uncertainty (SU) matrix. The problem is that some variables have infinite entropy (H) value which results in zero SU. How can I handle this entropy value?
what do you mean pairwise MI matrix?!?!
what is matrix A? is it joint probability distribution of random variables X and Y?
What is the type of the matrix A values ?
Usually the mutual information using the matrix with integer values, works well.
May be you need to normalize your matrix values.
Entropy cannot be infinite, by definition. It is log(p)*p, which equals zero if p=0. When you implement this in code, however, you need to ensure that the computer understand this. I usually do this by using log(p+eps)*p, where eps is a very small number.
The data you have provided is insufficient. For a discrete random variable X, which takes value x1, x2 ,... with pmf p(x1) , p(x2), ... entropy is defined as H(X)=-sum p(xi)log(p(xi)), p(xi) has maximum value 1, and min value 0. when it is 1, that term become 1log(1)=0. Now when it is 0, the corresponding entropy term becomes -0log(0), which can be misunderstood as infinity, but this is handled in this way. we calculate limit p->0 -plog(p) =0 , hence we define 0log(0)=0. This is called removable discontinuity in entropy function. Hence entropy for such case is define as H(X)= -plog(p) if p not equal to 0, else H(X)=0 if p=0. I think this helps. It is better you give more description of the problem.
Entropy: A concept that is not a physical quantity
https://www.researchgate.net/publication/230554936_Entropy_A_concept_that_is_not_a_physical_quantity
I think this code will solve your problem. I checks that the probability is non-zero before calculating the entropy: https://www.mathworks.com/matlabcentral/answers/27235-finding-entropy-from-a-probability-distribution
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Probability
- Probability Theory