The rfe functions in the caret package allow to perform recursive feature selection (backward) with cross-validation.
It is expected that the best features selected in each fold may differ, as also stated in the caret webpage (http://topepo.github.io/caret/recursive-feature-elimination.html)
"Another complication to using resampling is that multiple lists of the “best” predictors are generated at each iteration. At first this may seem like a disadvantage, but it does provide a more probabilistic assessment of predictor importance than a ranking based on a single fixed data set. At the end of the algorithm, a consensus ranking can be used to determine the best predictors to retain."
However it is not clear to me how the final "best" set of predictors is chosen in rfe, considering this expected heterogeneity among folds. I cannot find the procedure of the "consensus ranking" mentioned above.
Thank you for you help!
Dear Massimiliano,
I use RFECV provided by the scikit-learn package in Python, but it presumably works the same way as the corresponding method in the R's caret package.
RFE with cross validation works approximately this way. You have to specify as input: a classifier C you want to use for prediction (e.g., a decision tree), a scoring function F to evaluate the performance of the classifier (e.g., accuracy); how many features k you want to eliminate in every step (e.g. 1 feature). Then RFE with cross validation starts with all the n features, makes predictions with cross validation using C, computes the relative cross-validated performance score F and the ranking of the importance of the features. Then it eliminates the lowest k features in the ranking and re-makes the predictions, the computation of the performance score and the feature ranking. It proceeds until all the features are eliminated. Finally it outputs the set of features which produced the predictor with the best performance score.
See also the paper at the second link.
Regards,
Luca
http://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html
http://clopinet.com/isabelle/Papers/geneselect.pdf
I develope 100-fold cross validation method. And the best model with minimum mean of error rate in the validation samples is the best procedure for feature selection.
See my papers at RG.
Dear Massimiliano,
I use RFECV provided by the scikit-learn package in Python, but it presumably works the same way as the corresponding method in the R's caret package.
RFE with cross validation works approximately this way. You have to specify as input: a classifier C you want to use for prediction (e.g., a decision tree), a scoring function F to evaluate the performance of the classifier (e.g., accuracy); how many features k you want to eliminate in every step (e.g. 1 feature). Then RFE with cross validation starts with all the n features, makes predictions with cross validation using C, computes the relative cross-validated performance score F and the ranking of the importance of the features. Then it eliminates the lowest k features in the ranking and re-makes the predictions, the computation of the performance score and the feature ranking. It proceeds until all the features are eliminated. Finally it outputs the set of features which produced the predictor with the best performance score.
See also the paper at the second link.
Regards,
Luca
http://scikit-learn.org/stable/auto_examples/feature_selection/plot_rfe_with_cross_validation.html
http://clopinet.com/isabelle/Papers/geneselect.pdf
- Badges
- Science topic
- Similar topics
- Medicine
- Physiology
- Biosignals
- Biomedical Signal Processing
- Statistical Analysis