The primary purpose of the Information Gain is to determine the relevance of an attribute and thus its order in the decision-tree. An attributes (variable) with many distinct values, the information gain fails to accurately discriminate among the attributes.

Dear Gulzar Shah ,

Thanks for your answer. So, it cannot work well for features with discrete values, right?

No. It does not work good for attributes with large number of distinct values (over fitting issue) [1]. To solve this issue the information gain ratio has been proposed. However it has its own problem as well[2]. For a comprehensive view about feature selection methods, their advantages, and disadvantages please refer to [3].

Best regards,

Reza Sadeghi

Reference:

[1].https://www.slideshare.net/marinasantini1/lecture-4-decision-trees-2-entropy-information-gain-gain-ratio-55241087

[2].https://www.quora.com/What-are-the-disadvantages-of-using-Information-Gain-Ratio-as-a-metric-for-building-decision-trees

[3].https://pdfs.semanticscholar.org/310e/a531640728702fce6c743c1dd680a23d2ef4.pdf

why Information gain is biased toward choosing features with a large number of values. I can't prove

Please note that information gain (IG) is biased toward variables with large number of distinct values not variables that have observations with large values. Before describing the reason of this condition, lets review the definition of IG.

Information gain is the amount of information that's gained by knowing the value of the attribute, which is the entropy of the distribution before the split minus the entropy of the distribution after it. The largest information gain is equivalent to the smallest entropy.

In other words, a variable with the highest number of distinct values probability can divide data to smaller chunks. Also, we know that lower number of observations in each chunk reduces probability of variation occurrence.

Using ID variable in splitting data is a common example for this issue. Since each individual sample has their own distinct value, selecting ID features leads to many clusters with one sample and entropy of zero. Therefore, a decision tree that works with IG, selects the ID as the first separator attribute. Indeed, entropy will approach to zero by selecting the ID feature. However, we are not interested to such a feature. We are more interested to features that highly explain the variation of dependent variable.

What is the disadvantage of using Information Gain for feature selection?

-->Natural bias of information gain: it favours attributes with many possible values.

-->Consider the attribute Date in the PlayTennis example.

-->Date would have the highest information gain since it perfectly separates the training data.

--> It would be selected at the root resulting in a very broad tree

--> Very good on the training, this tree would perform poorly in predicting unknown instances. Overfitting.