Ultimate challenge in such tasks is actually robustness; and to achieve robustness you can select optimum number of feature sets; rather than features.

What exactly do you mean with optimal number of feature sets? Should I think about ensembles here?

Hi Jean,

There is no magic number that works for all the tasks, simply you can evaluate the selected features based on classifier performance using those features. According to that, you can increase/decrease the number of selected features. 

HTH.

Samer

Hi Samer, thanks for the answer. I strongly agree with you, yet, several papers still use some "magical" numbers to select the number of features. My idea would be to find a proper rationale for these specific numbers or if they are rule of thumbs with no solid rationales behind them.

Dear Jean,

In fact, it is not an easy mission because of several reasons: number of the original features (loaded/utilized) changes from task to another. In addition, nature of those features plays a significant role in determining the selected features number. Finally, relation between the features, from the one hand, and the utilized algorithm for feature selection, from the other hand, are another factors to be aware of during this process.

HTH.

Samer

Dear Jean Paul,

From my point of view, this problem area does not have an exact answer.

Goal is to find a trade-off between the accuracy and the execution time (computational cost).

The execution time is directly connected to the number of features used.

Hence, our problem is an optimization problem where intention is to optimize the solution in such a way that we can have the highest accuracy together with the lowest execution time (smaller number of features).

Feature selection is aimed to reach to this point of equilibrium.

Best

Peyman 

I think this is quite subjective. When I do principle component analysis to get the main features of a large dataset. I usually use the first several components which explain 80% or 90% of the whole variance.

I agree, it is quite subjective. But, PCA is a good suggestion

Hi Jean,

I have utilized an evolutionary optimization methodology to find the optimal number of features to be selected (the link attached). It is though for gene selection, but is also applicable to general feature selection.

Majid 

Article Robust and stable gene selection via Maximum–Minimum Corrent...

I have a question...is using SelectKBest to select 50% of the features in a dataset an optimal way in feature selection? I tend to select my features using this method, but it is the right way of doing it?

Hi,

If the computational time does not matter and you want to obtain the highest accuracy, one possible approach is to consider a loop, in which the features are added to the prediction model gradually (one by one) based on their importance weight. Subsequently, the optimal feature set can be considered a set leading to highest prediction accuracy.

You could perhaps take a look at autoencoders for advanced feature selection: Preprint Fractal Autoencoders for Feature Selection

Cheers, Raoul

Hey Jean,

I usually plot the training and test error (as obtained in a cross validation) with respect to a decreasing number of features used to train the underlying model. Your training error should increase constantly with decreasing number of features. Your test error should decrease up to a certain point before it increases again. This identified minimum, the sweet spot, defines the optimal number of features.

I hope this helps!

Cheers,

TIm

You can use RFECV.