Dear all,
Conceptually, what the difference between mutual information and information gain?
I have tried to do an exercise which is involved the mutual information theory, However, after I ended up the calculation for MI and tested as well using the Information Gain theory by referring the same figures from a sample table, i got the same value between MI and IG. Can anyone here brief me the keyword to differentiate these both theories?
in the context of feature selection, both measure how relevant a feature to a specific class, yet while mutual information measures the positive features only, information gain measures both negative and positive features. Therefore, mutual information is one sided metric whereas information gain is two sided metric.
I've seen different sources using the term information gain to mean either MI or KL-Divergence. You should look into how IG is defined, mutual info I(X;Y) = H(Y) - H(Y|X) while the KL Divergence D(X||Y) = H(X,Y) - H(X). But from the description you are , it looks like IG might be being used interchangeably with MI.
Again, double check the definition of your sources to see if this is the case.
I hope that helps.
Cheers
I can explain information gain, however I am not sure If you are using it in the same context as I was doing.
Information gain is the measure of entropy gained due to the operations you perform on a given data/random variable. The gain is measured in terms of entropy. Entropy is essential the measure of variation in the data, the lesser the variation you have the lesser the entropy is and the greater the data is correlated.
Consider the example of determining a split of data into two subset which maximizes the information gain. This in turn means that newer subsets will contain data which is highly correlated within each subset as compared to the data as a whole. In such a scenario the first equation in the following link is used:
https://en.wikipedia.org/wiki/Information_gain_in_decision_trees
A very good example of this can be found here: http://www.cs.csi.cuny.edu/~imberman/ai/Entropy%20and%20Information%20Gain.htm
I know this is a half answer since I have no idea about mutual information. Hoping this can be put into context for better understanding.
in the context of feature selection, both measure how relevant a feature to a specific class, yet while mutual information measures the positive features only, information gain measures both negative and positive features. Therefore, mutual information is one sided metric whereas information gain is two sided metric.
thank u francisco, Muhammad asad and Anwar ali. But for Anwar ali, which one technique is better to be applied in feature selection from your point of view?
Actually they are the same. IG of X given Y is given by IG(X|Y)=H(x)-H(X|Y), and Mutual info I(X;Y) is given by I(X;Y)=sum_x sum_y P(X,Y) log {P(X,Y)/P(X)P(Y)}. If you manipulate I(X;Y), you eventually ends up having I(X;Y) = H(X) - H(X|Y) = IG(X|Y).
- Badges
- Science topic