Hi,
For doing the evaluation of different feature subsets in the wrapper feature selection methods we need to use the train and the validation sets. and after selecting the best feature subset we use the testing set wich is different from the validation set in order to get the final performance.
We split the dataset randomly in three subsets : training (70%), validating (20%) and testing (10%).
After getting the best feature subset using the training and the validating sets , We evaluate this feature subset on the testing set and we found bad performance. We have tried this for several randomly chosen divisions.
in general we have observed that the feature subset that gives good performances (accuarcy) for the testing set gives inferior performance for the validating set. so the question is how to carfully choose the validation and the testing sets in some manner to get the best feature subset for the testing set but using the validation set.
Thank you.
Why do not you try to change the percentage of the training, validation and testing sets to 60%, 20% and 20% and check if your error is decreased. Sometime the high error in the validation phase is due to over training.
it is a data set (for example lonosphere from the UCI repository) that was devided randomly into three subsets so for us the difference is unkonw juste for the number of instances in each sub set and may be also the number of instances in each classes. in some work, authors take 70% for training , 20% for validating and 10% for testing. since this portion are fixed I think we can not interchange.
Dear Wassila Guendouzi,
I agree with Georgios Leontidis that "The best way is to either run a cross-validation and choose the K-fold depending on the amount of data" you can refer to the paper "
"Shear Strength Prediction in Reinforced Concrete Deep Beams Using Nature-Inspired Metaheuristic Support Vector Regression" and refer some useful cited reference from this paper to know "a cross-validation method".
Please contact me if you have further question.
"
Hi Wassila,
The problem when you divide your data in training, validation and test sets is that you may have a "lucky" partition, so as other mentioned the best way is to perform a nested-cross-validation. With this approach you will have different partitions of training, validation and test sets and at the end you get the average performance which may be a more realistic one.
You can see an example in the following link, where they implement a svm-rfe feature selection algorithm. https://github.com/johncolby/SVM-RFE
Regards,
Andres
Dear Wassila,
Use leave-one-out method (see https://www.researchgate.net/publication/221184961_Tournament_Searching_Method_to_Feature_Selection_Problem where I also used tournament searching as wrapper). Or use local version of leave-one-out: use only the most similar training samples to the test samples to leave in LOO.
Conference Paper Tournament Searching Method to Feature Selection Problem
Dear Wassila,
I would definitely advice you to use K-fold cross validation approach, as with single partition you may have "lucky" partition as Andrez already mentioned.
What you need to remember is that there are two approaches for wrapper feature selection methods.
First one is called Non Recursive Feature Elimination (NRFE) and in this approach the variable ranking is computed only once at the beginning of the learning process. Next, less important variables are removed from the ranking and the random forest is learned based on the remaining set of values. This step is repeated in several iterations until no further variables remain. Second approach is called Recursive Feature Elimination (RFE) and it differs from NRFE method is such a way that the importance ranking is updated (recomputed) at each iteration. Then, similarly to NRFE, the less important variables are removed and random forest is learned. There are some extensive simulation study was performed comparing these two approaches showing that RFE approach as it might be more reliable than NRFE since the ranking by the permutation importance measure is likely to change at each step and by recomputing the permutation importance measure we assure the ranking to be consistent with the current model.
The above is a citation from my paper which is just about to be published on the ICMMI Conference Proceedings (http://www.springer.com/us/book/9783319234366).
If your are interested in more details, or reading my paper, please contact with me. The paper will be also available on my RG profile as soon as it is published on Springer website.
You can use a cross-validation approach.
The performance with cross-Validation should be better because this algorithm's performance takes into account the situation where there is unseen instances, while using all the available data.
What type of cross-validation you should use depends on your data. Some popular variations are K-Fold Cross-Validation, Leave one out Cross-validation and Stratified Cross-Validation.
You can divide all your data in 2 subsets... one you use to perform cross-validation and find the best feature selection, and the other will be used to assess the final performance.
- Badges
- Science topic