There are mainly three different groups of FS techniques: filter, wrapper and embedded. For example PCA belongs to filter methods, GA is a wrapper, and SVM-RFE or Random Forest are examples of embedded approaches. I recommend you to read doi:10.1093/bioinformatics/btm344 and http://dx.doi.org/10.1016/j.eswa.2011.08.051

Good question.

Two feature selection not yet mentioned are given next.

concave minimization: generate a separating plane by minimizing a weighted sum of the distances of misclassified points to two parallel planes that bound the sets and which determine the separating plane midway between the set of misclassified points.

support vector machine approach: minimize the weighted sum of the distances between misclassified points to the bounding planes AND maximise the distance between the two bounding planes that generate the separating plane.

For more about this, see the attached pdf file.

There are mainly three different groups of FS techniques: filter, wrapper and embedded. For example PCA belongs to filter methods, GA is a wrapper, and SVM-RFE or Random Forest are examples of embedded approaches. I recommend you to read doi:10.1093/bioinformatics/btm344 and http://dx.doi.org/10.1016/j.eswa.2011.08.051

There are many approaches which are based on spectral clustering for unsupervised feature selection.

Though feature selection and feature transformation are sometimes confused, the optimal feature selection method is the exhaustive search where all possible feature combinations are evaluated. Since its computational burden is quite high, suboptimal wrapper techniques such as forward/backward selections, GA, etc. are preferred by researchers.

There are good sparse Bayesian methods using expectation maximisation, and the sparse Bayesian version of SVM, the relevance vector machine RVM.

Another feature selection not yet mentioned is the boosting approach: select the "best" feature in each round which optimizes some criterion using a distribution of sample weights that are updated so as to give more importance to the samples not matching the criterion. Some examples of criteria are i) minimizing the exponential loss (AdaBoost, GentleBoost, etc.) and ii) maximizing the area under the ROC curve (RankBoost).

Even though it is possible to directly apply a boosting method for feature selection, I would suggest a work which efficiently decouples feature selection from training and it uses an AdaBoost-based feature selection method:

- Wu, J. et al. Fast Asymmetric Learning for Cascade Face Detection. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE. 2008

There are also many other works which describe the use of boosting for feature selection, e.g:

- Redpath, D.B. and Lebart, K. Boosting Feature Selection. Pattern Recognition and Data Mining. 2005

Hope that helps.

As usual, the answer entirely depends on your problem. Are you looking for improved performance (for instance in case of limited training data availability) or the identification of relevant input feature (which can be used to raise hypothesis or gain understanding).

For instance, RVM (Tipping 2001, Sparse..) can infer hyperparameters indicating the relevance of each input function, which however sometimes comes at the cost of decreased performance!!! The same is true for Automatic Relevance Determination (McKay 1992) for Neural networks.

Also, I would personally classify Principal Components Analysis as a "dimensionality reduction" technique rather than a "feature selection technique". This being said, it is a completely valid approach if you're trying to "concentrate" your data variability within a few axis. Beware that if the variability is cause by noise you may loose the relevant information.

If you're not sure, I would recommend Relief (Kira, 1992 practical) which offers a good trade-off between complexity (it's extremely easy to understand what's going on with this one) and performance.

Finally, different fields usually prefer different types of feature selection techniques, partly due to the nature of data present in the field, partly by habit. Check literature in your field to see what has work the best on your type of data and go for it... it's always the best way to start and it will give you a baseline to compare your results with in case you want to try something else.

PCA is more a feature reduction method rather than feature selection. Moreover, the final subset is projected on a different subset space so it is not interpretable.

There is really no "best" method for feature selection because different data sets have different measures of correlation. PCA is a dimension reduction technique (than direct feature selection) which creates new attributes as a combination of the original attributes in order to reduce the dimensionality of the dataset and is a univariate filter method. CFS is a multivariate filter method that favours features which are correlated with the classification but uncorrelated with each other. There are other methods such as the LASSO, SVM-RFE (which Carlos mentioned), Regularized random forests (RRF) and RFRFE. There performance vary with different kinds of dataset though I got overall higher performance from the SVM (in terms of accuracy and sensitivity) than other methods such as Naive bayes, and decision trees.

Have a look at the attached file as well as other materials.

PCA and PLS will degrade if a large number of low relevance or irrelevant descriptors or features and present. The sparse Bayesian methods set low relevance features identically to zero and avoid this issue, and also make models easier to interpret.

As has been stated already, there are several types of features. A toolbox written in Java covers the most important selection methods. It can be used as a plugin to RapidMiner. Software and documentation (a nice overview of methods) is here:

http://sfb876.tu-dortmund.de/auto?self=$ddwpqancao

For extraction of features you might also want to look at:

http://vista-tv.eu/

Nikolay Zagoruiko, Sobolev Institute of mathematics SD RAS, Novosibirsk, Russia

Wavelet analysis is another approach with multi- resolution principle. Basically, coarse to fine step by step feature filtering is roburst.

As very rightly pointed out by previous answers, PCA is not a feature selection technique. It is a dimension reduction technique: it provides a compact representation of the candidate variables, irrespective ot the quantity that the model is supposed to predicted. If the candidate variables are irrelevant for predicting the quantity of interest, PCA provides a compact representation of irrelevant variables, which is not very helpful for designing a model. A resonable method consists of first using a real feature selection method (filters are usually less computer-intensive than wrappers), and subsequently using PCA on the selected features for further dimension reduction. The book by Guyon et al. "Feature Extraction, Foundations and Applications" (Springer) contains essential basic information on the topic.

A recent trend in feature selection is to use deep learning methods. Andrew Ng from Stanford has done a lot of work in the area. Take a look at his page and also on a recent workshop organized last year at UCLA on feature learning:

http://ai.stanford.edu/~ang/papers.php

http://www.ipam.ucla.edu/programs/gss2012/

Also take a look at this robust feature selection method for audovisual emotion recognition using deep learning:

http://web.eecs.umich.edu/~emilykmp/EmilyPapers/KimICASSP2013b.pdf

Hi,

as many colleagues already pointed out, there are several different feature selection methods which can be broadly classified into filter, wrapper, and embedded approaches. The choice of the best approach is highly problem dependent, but in general you will have to trade off speed of execution (i.e. computational time to execute the feature selection process) for performance (i.e. how well the system performs the intended task using the reduced set of features). Filter approaches are the fastest to execute, but the final system performance may not be optimal, wrapper approaches usually guarantee high system performance but are also computationally expensive, embedded approaches tend to fall somewhere in between the two above approaches.

Also, as already pointed out by other contributors in this thread, PCA is not a feature selection method but a feature extraction technique. It creates a smaller set of new features as a combination of the old ones, but the new features may not have a clear interpretation.

If you are interested, I have published a couple of experimental papers on comparisons of various feature selection approaches (PCA, ReliefF, Evolutionary wrapper and embedded methods, OCD). You should find them for download all in my profile.

Castellani, M. and Rowlands, H. (2008), Evolutionary Feature Selection Applied to Artificial Neural Networks for Wood Veneer Classification, International Journal of Production Research, vol. 46, no. 11, pp. 3085–3105.

Castellani, M. and Marques, N.C. (2008), FeaSANNT - an Embedded Evolutionary Feature Selection Approach for Neural Network Classifiers, VIMation Journal, no. 1, pp. 46-53.

Castellani, M., (2013), Evolutionary Generation of Neural Network Classifiers – An Empirical Comparison. Neurocomputing vol. 99, pp. 214-229.

Yes, deep learning methods are necessary for feature selection, and we must resolve the hard problems such as combinatorial explosion　in feature matching.

No, what you mentioned are not recent, but widely used. you can browse thro the net for recent development.

Obviously PCA is an error in my previous comment, it is a feature extraction technique. To avoid errors I recommended two papers which clearly summarize features selection techniques.

Definitely there is no best or recommended feature selection method. Feature selection implies a process in which some methods should be tested and compared. But of course, any wrapper or embedded method with the classifier of choice shall be used as it might, in most cases, be specific for that particular model. I recommend the use of the software WEKA, in which several feature selection methods has been implemented.

Nonlinear kernel-based learning was left out.

A very nice book of standard methods of feature selection is:

Huan Liu, Motoda, Hiroshi, "Feature Selection for Knowledge Discovery and Data Mining" Springer ,1998

It is not new but very simply describes the approaches to feature selection and shows example algorithms.

Yes, there other feature selection methods commonly used in image processing, microarray data any many other domains where feature selection is necessary. These are, SVM-RFE, SNR, Rough set, fuzzy rough set and kernelized fuzzy rough set.

D. Chakraborty

You could do feature selection and reduction of dimensions with self-organizing maps.

You can read my paper for this matter

Article Feature Selection in Imbalance data sets

The choice of feature selection method depends on the classifier to be used and on the application, e.g.: Random Forests-based selection is likely to provide the feature set best performing with the Random Forests classifier, whereas F-score-based selection will probably select a feature set best performing on the same data with the Soft-margin SVM classifier, which is the core of the F-score selection strategy. Nevertheless, none of these solutions will reedy do a good job with highly imbalanced problems such as anomaly detection. Here I agree with the suggestion by Mohammad Bazmara. You can also read

Conference Paper Learning from Only Positive and Unlabeled Data to Detect Les...

And this

Conference Paper Feature Selection for SVM-Based Vascular Anomaly Detection

Nikolay Zagoruiko, Sobolev Institute of mathematics SD RAS

I think MRMR (Maximum Relevance Minimum Redundancy based on mutual information) is a good one.

Feature selection methods can be classified into supervised or unsupervised. If class label is available, supervised feature selection is appropriate; otherwise unsupervised feature selection approach can be applied. Though PCA and Genetic based methods are applied for feature selection, Rough set based feature selection methods provide good results for many data sets.

You can also try social impact theory based optimizer and opinion dynamics based optimizer for feature subset selection.

You can read the papers at the following links:

http://dx.doi.org/10.1016/j.neucom.2013.03.063

http://www.nature.com/srep/2013/131021/srep03008/full/srep03008.html

These are wrapper based approaches.

Linear Discriminant Analysis (LDA) can be used for feature selection technique in place of PCA, which got lot of popularity in recent years. See the below link ..

http://research.cs.tamu.edu/prism/lectures/pr/pr_l10.pdf

I think the sparse regression methods, such as, elasticnet, lasso and fused lasso as well as sparse PLS are also useful for many kinds of data set. In addition there sparse versions of PCA that can be used to do feature selection.

Yes sparse methods are often very useful especially with data sets with thousands or tens of thousands of independent variables. We use multiple linear regression with an expectatation maximisation algorithm using a Laplacian prior. A related approach can be used with neural networks regularized by a sparse prior to effect nonlinear feature selection. We adapted these from Mario Figuieredo's work. See Le et al. Chem Rev 2012

Results of comparison of methods of a choice of attributes can be seen in this article.

Here is a list of feature selection algorithms with Matlab implementation. Fisher Score and mRMR are fast and gives good results.

http://featureselection.asu.edu/software.php

Dear Farooq, your answer is very good. I think that I will learn much from given address. Thank you very much!

A review of feature selection methods based on mutual information

Jorge R Vergara, Pablo A. Estevez

Neural Computing and Applications

DOI 10.1007/s00521-013-1368-0

Available online 2013 (Springer)

Some examples can be found in:

1. Feature selection: Evaluation, application, and small sample performance, by A Jain, D Zongker

2. Rough set methods in feature selection and recognition, by Roman W. Swiniarski and Andrzej Skowron.

Some great answers here. It's not clear to me what level of detail you ultimately need. For example, SIFT (Scale Invariant Feature Transform) and SURF (Speeded Up Robust Features) are common feature methods with many variations (RIFT, G-RIF, PCA-SIFT). Are they state-of-the-art though? Well, the earlier versions probably are not and the later ones may be. Also, these are specific algorithms and the examples in your question seem to be more about fundamental mathematical basis of techniques.

Are you looking for specific algorithms or generalized mathematical approaches? Also, are you looking for "state-of-the-art" in a literature search context, or actual performance in real-world application? I think these details matter in helping answer the question.

Cheers.

Please look into it, Some Methods are discussed in detail

http://iajit.org/PDF/vol.10,no.6/4134.pdf

see this:

Yifeng Li, Chih-Yu Chen, Wyeth W. Wasserman:

Deep Feature Selection: Theory and Application to Identify Enhancers and Promoters. RECOMB 2015: 205-217

https://www.researchgate.net/publication/301691146_Feature_Selection_Library_MATLAB_Toolbox

Source Code:  

https://it.mathworks.com/matlabcentral/fileexchange/56937-feature-selection-library

Technical Report Feature Selection Library (MATLAB Toolbox)