Hi,
Please I need to kow how can i use cross validation to evaluate a wrapper fetaure selection method.
I already do this by spliting my data sets into three different sets :
training set , validating set (to evaluate every feature subset during the feature selection method since it is a wrapper method ) and the test set (used to do the evaluation with final extracted features).
I konw that the three sets must be different.
How can I apply cross validation in this contexte in order to avoid the overfitting and do a correcte evaluation?
for example can I use the training set and the validation set to evalutes the feature subsets and then apply the cross validation on the same training set for the finale evaluation (final extracted features ) !!!??.
Hope My question is claire enough ?.
Thank y very much.
The training/validation sets should be different than the test set that is used to evaluate the final performance of the feature set and classifier. Your suggestion about using the training/validation sets to select the features and then performing cross-validation on the same data to calculate the final performance should not be done. This is a subtle form of training on the test set since feature selection should be viewed as a training process. The performance estimated from this procedure is possibly optimistically biased (though I'm not sure how bad the performance estimate will be).
There could be multiple correct ways of performing this type of feature selection, but I'll describe one way here. This will perform forward selection based on maximizing cross-validation accuracy.
1. I assume our data set is already split into disjoint training and test sets. If this is not already done, we could just split the data randomly into training and test.
2. Split the training set into cross-validation sets.
3. Choose the first feature that has the highest cross-validation accuracy.
4. Continue to add features one-by-one based on maximizing the cross-validation accuracy.
5. Terminate the feature selection once a stopping criterion is reached.
6. Train a final classifier using the selected feature set and the full training set.
7. Test the final classifier on the held-out test set, which was not used at all for feature selection or classifier training.
RFECV does not have good documentation so I am not exactly sure what it's doing. It appears to do something similar to my proposed method. It splits the training data into cross-validation sets, uses the RFE criterion to rank features (for linear models, rank the features based on the absolute value of their weights in the classifier), then it measures the cross-validation accuracy as a function of the number of features added in the model. Other than the different selection/ranking criteria, I don't really see how this is different than my proposed method. This would be a useful function to use if RFE is the desired selection criteria.
- Badges
- Science topic
- Similar topics
- Correction