I created classification tree using rpart package with R? The tree model consists of 5 variables ( I used totally 10 variables). However, when you look at the variable importances, it seems that the variables that are not included in the tree are also important (Variable importance values are given for all 10 variables). What could be the reason for this situation?
Dear Serkan Özdemir
The entropy is a measure of homogeneity. When the homogeneity sample is completely homogenous, the entropy is 0; otherwise, the entropy is 1. All of the other alternatives are erroneous since they are unrelated to the provided circumstance.
A tree looks at one feature at each node to locate a split point in order to create the best leafs possible. It determines a split point by taking into account that every node carries information, and the tree's purpose is to maximize the information "gain" existing at that node (called impurity measures).
Kind Regards
Qamar Ul Islam
Serkan Özdemir ,
This is a common occurrence especially when you have more features than there are splits in the tree. Now, a tree is just a set of partitions of the feature space that are mapped to the target space. While building this tree, the algorithm evaluates many different splits over different features (or variables) at each stage. Only the best split is used to build the tree. However, the information from this evaluation is used in determining feature importances.
Let's step through this with a simple example: Let's say that I have a dataset with 3 binary explanatory features (variable A, variable B, Variable C) and one binary target. I just want a stump (ie, one split leading to a simple tree with one root and 2 leaf nodes). While determining the best feature for this split, the algorithm will evaluate all three features to see the decrease in impurity it gets by splitting based on the value of variable A, variable B, and Variable C.
Let's say that the decrease in impurity by splitting on Variable A is 0.5,
by splitting on Variable B is 0.2, and
by splitting on Variable C is 0.05.
With this evaluation, the algorithm will choose to split based on Variable A and that single split will define your tree. So, your final trained tree will have only 1 split based on Variable A.
However, the algorithm will have the decrease in impurity by splitting on all three features, that it will use to define feature importances. In this case, the ordering may be Variable A > Variable B > Variable C, as the decrease in impurity is maximum if the split is based on the value of A, and minimum if the split is based on the value of variable C. So, in this toy illustration, you can see that even though variables B and C are not used in defining the final tree, their relative feature importances have been generated due to the evaluations that the algorithm did while constructing the tree.
- Badges
- Science topic
- Similar topics
- Philosophy
- Philosophy Of Science
- Reasoning