In a simple experiment, I have a dataset with 200 samples, each includes 5 attributes and a binary class label. 3 of these attributes are supposed to have equal measures for all samples in the dataset (we suppose there are 3 fixed numbers). Now I classified the dataset by random forest using R but the result is strange.
Having these 3 attributes in a dataset, classification is going wrong and all the samples take just 1 label (say label A) .
Eliminating these 3 fixed attributes, classification is correct with more than 96% accuracy.
The question is: what is the reason for this different result?
What is the effect of eliminating some fixed equal attributes on the classification problem?
It is a good practice to eliminate the attributes which have a constant value for all the samples in your dataset.
There are several reasons for justifying this, which mainly depend on the classification algorithm you are using.
For example if you call you data matrix X (samples x features) and you have a constant feature the matrix X'X will have reduced rank and therefore algorithms that make use of this matrix (e.g. least squares regression) may not find a solution or may run into numerical problems.
The I guess that in the training of a decision tree (a random forest is just an ensemble of decision trees) this might cause problems in the selection of the split attribute. On which function of your dataset do you perform the splits? Entropy? I guess the presence of constant attribute may cause numerical problems in the calculation of the function on which to base the split, but this is pure speculation (I don´t know the R function for random forests in detail).
Hope this helps!
Cheers
@Marco
Thank you. It was helpful. So, do you think elimination of fixed attributes from a dataset has no negative effect on classification?
- Badges
- Science topic