I've seen different sources using the term information gain to mean either MI or KL-Divergence. You should look into how IG is defined, mutual info I(X;Y) = H(Y) - H(Y|X) while the KL Divergence D(X||Y) = H(X,Y) - H(X). But from the description you are , it looks like IG might be being used interchangeably with MI.

Again, double check the definition of your sources to see if this is the case.

I hope that helps.

Cheers

I've seen different sources using the term information gain to mean either MI or KL-Divergence. You should look into how IG is defined, mutual info I(X;Y) = H(Y) - H(Y|X) while the KL Divergence D(X||Y) = H(X,Y) - H(X). But from the description you are , it looks like IG might be being used interchangeably with MI.

Again, double check the definition of your sources to see if this is the case.

I hope that helps.

Cheers

I can explain information gain, however I am not sure If you are using it in the same context as I was doing.

Information gain is the measure of entropy gained due to the operations you perform on a given data/random variable. The gain is measured in terms of entropy. Entropy is essential the measure of variation in the data, the lesser the variation you have the lesser the entropy is and the greater the data is correlated.

Consider the example of determining a split of data into two subset which maximizes the information gain. This in turn means that newer subsets will contain data which is highly correlated within each subset as compared to the data as a whole. In such a scenario the first equation in the following link is used:

https://en.wikipedia.org/wiki/Information_gain_in_decision_trees

A very good example of this can be found here: http://www.cs.csi.cuny.edu/~imberman/ai/Entropy%20and%20Information%20Gain.htm

I know this is a half answer since I have no idea about mutual information. Hoping this can be put into context for better understanding.

in the context of feature selection, both measure how relevant a feature to a specific class, yet while mutual information measures the positive features only, information gain measures both negative and positive features. Therefore, mutual information is one sided metric whereas information gain is two sided metric.

thank u francisco, Muhammad asad and Anwar ali. But for Anwar ali, which one technique is better to be applied in feature selection from your point of view?

Actually they are the same. IG of X given Y  is given by IG(X|Y)=H(x)-H(X|Y), and Mutual info I(X;Y) is given by I(X;Y)=sum_x sum_y P(X,Y) log {P(X,Y)/P(X)P(Y)}. If you manipulate I(X;Y), you eventually ends up having I(X;Y) = H(X) - H(X|Y) = IG(X|Y).

I agree from that definition they are the same, but since they are symmetric, the notation IG(X|Y) could be confusing since it reminds probability notation. I would prefer using IG(X,Y) or IG(X;Y) as it is done in mutual information.