Salem,
In a wrapper feature selection algorithm, we use the accuracy rate to evaluate the candidate feature subsets during the search process.
How can we show that a wrapper feature selection algorithm hasn't overfitting.
can we use the final selected features (feature subset) to calculate the accuracy on the training set and the validating set, and then compared the two obtained accuracies. According to how much closer the two accuracies are, we decide about the overfitting.
Thanks.
It is an interesting question. I have found to prevent overfitting with a CGP-based generative hyper-heuristics is to use the relative standard deviation. If it increases when I learn with an easy, medium and hard instances then it is unlikely to overfit.
It is worth trying.
Hi, in wrapper approach you may want to use learners that are not prone to overfitting (e.g. support vector machines). Then, it is also a good practice to log what wrapper does internally by plotting each validation set performance against each trial subset in a scatter plot so that in the end you can see that your final selected subset really performs best on unseen (validation) data.
Cheers
P
Thanks patricia Ryser-welch and Pekka Jounela
if i use a scatter plot to see the validation set perfromance agaist each trial subset. should i compare it with the one of the traing performance.
thanks
Hi, there is no need to show training set performances which refer to overfitting i.e. poor generalization. Validation set performance is of primary interest because it refers to generalization.
See for example page 69 in Statnikov et al. 2009: A Gentle Introduction to Support Vector Machines in Biomedicine.
Cheers
P
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence