If you have the time, it actually is very logical to try every combination of the feature selection (FS) and classification methods. Each classier method is limited by the choice of its training algorithm and how well that algorithm is implemented. The same comment applies to FS. The probability that a given person has the best code for each classifier and FS method is close to zero.

In theory, you can reduce the amount of necessary work because:

(1) Many available methods (for both feature selection and classification) are related to the Bayes classifier concept.

(2) The classification technologies have different different capabilities and complexities.

The k-NN, ANN, RBF, FLN (functional link net), PLN (piecewise linear network(s)), and Bayes-Gaussian classifier all converge to Bayes classifiers under certain conditions.

Feature selection methods mostly employ filter or wrapper subset evaluation functions (SEFs). Wrappers are basically classifiers, and can be some of the same ones that can converge to Bayes classifiers. Therefore, you can decouple the feature selection task and classifier choice task:

(1) Find many good feature subsets using a wrapper method that uses a Bayes-related classifier.

(2) Using the candidate subsets, evaluate Bayes-capable classifiers that match the number of classes and the amount of training data that you have. Or, try all of them.

I think for classification, the question is more like whether the classifier would identify discriminating set of features by itself. If the answer is no or somewhat not sure, then a feature selection could help, provided the feature selection technique is robust on that problem and it does not remove the useful features. On the other hand, certain classifiers would not prefer large number of features (like Neural nets and SVMs: reason computationally expensive), and in such cases also feature selection after a feature extraction would be useful.

Dear Alex,

yes! it is generally necessary to use feature selection after feature extraction for classification. If a feature had no predictive ability, you will add noise if you do not remove this feature. However, testing the model is slightly more difficult after feature extraction. In order to avoid overfitting, you will absolutely need a set of "new" observations, not present in the training set, for assessing the accuracy of your predictive model.

Yes, feature selection after feature extraction is definitely necessary in many cases of classification. Sometimes after feature extraction we may get many features which may not be much varied from each other and hence can be neglected. But discarding a feature which has definite importance in classifying that object may also effect predictive accuracy of that classifier. Feature selection may be particularly useful in cases where you have many features and sometimes using many features in a classifier may give problem. So for classifier you may have to discard some features.

Depends on the classifier and the problem. A nearest neighbor algorithm had different strengths and weaknesses than a decision tree method.

As mentioned above, feature selection can remove potentially remove features that are noisy (irrelevant) or that are highly correlated. Also, depending on the classifier, the more features, the more expensive the learning/training. Hence, while you may have features that are not correlated and not noisy, you may be better off with a subset of those that have similar predictive ability (and, again, at a computationally faster time).

In the case of noise, many methods assume (right or wrong), that a feature, if present, must be useful. Others attempt to weight features by importance; this can result in the 'elimination' of noisy features, but at greater computational cost in building the model. Plus, you are selecting (and preprocessing) a feature that is unnecessary.

For correlation, you may want to avoid highly correlated (again, exceptions occur). For example, having weight in centimeters, inches, and yards may not be very useful. Similar (same) information is being conveyed, even if there are small recording errors, but, in general, the classifiers will either need to learn to ignore all but one, or may assume that all three are important. In the latter case, height now becomes a major (and likely) unintentional influence.

As an aside, sometimes feature selection can remove the 'important' results. Your best bet is to try both (if feasible) and see what happens. But, a rule of thumb, is yes, do feature selection.

Yes

It depends on the application and any ground truth is available. if the ground truth is available. It is easy make a selection of features. In most case, features selected need be independent to each other

In my humble opinion, for sure, it is important to use feature selection, independing the application. Currently, there are two main problems related to the use of features that are still open. The first is: "is the set of features extracted enough to solve all vision tasks that I may have to do (or a vatiety of them)"? The answer could be: "there are still some features to be added to this set". So, feature extraction is still an open problem in computer vision. The second: "inside this set of features, which ones will I use to solve the current application problem?" A possible answer: "i could use a feature selector, depending on the application to solve this". Well, the set of features to be used would be learned or else defined empirically, based on the application. So this is, also, an open problem in CV. So, partial answers to these questions would probably push up the state of art in CV:-).

As others have already indicated, feature selection often helps, especially when the classifier being used is sensitive to noise or dependent features, as is the nearest neighbor classifier, Bayes-Gaussian classifier, neural network classifier etc.

Using the correct feature selection method is also important. Methods that evaluate each feature by itself are useless. Suppose that useful features for a given application are u and v, which are unavailable, and that the available features are (unknown to the user)

x1 = u + n,

x2 = u + n

x3 = v+2n

x4 = n

Here, the noise n is the same for each feature in a given training pattern. Some feature selection algorithms will report that x4 is useless.

A useful algorithm should report that the useful features are x1, x4, and x3, since they can be used, even by a linear classifier, to generate u and v as

u = x1 - x4

v = x2 - 2 x4.

I want to thank you all :)

also @ Ryan: you answer was perfect.

The other problem appears when we don't know which combination of classifier and feature selection method would be the best among several different methods (such as SVM, knn, ANN, ... as classifiers or mRMR and others as feature selection methods). although I prefer to use SVM as classifier and mRMR as feature selection method for classifying biosignal as EEG, EMG or EOG, ...)

Is there a rational procedure to select appropriate method to choose classifier and/or feature selection algorithm or we should use trial and error method?

If you have the time, it actually is very logical to try every combination of the feature selection (FS) and classification methods. Each classier method is limited by the choice of its training algorithm and how well that algorithm is implemented. The same comment applies to FS. The probability that a given person has the best code for each classifier and FS method is close to zero.

In theory, you can reduce the amount of necessary work because:

(1) Many available methods (for both feature selection and classification) are related to the Bayes classifier concept.

(2) The classification technologies have different different capabilities and complexities.

The k-NN, ANN, RBF, FLN (functional link net), PLN (piecewise linear network(s)), and Bayes-Gaussian classifier all converge to Bayes classifiers under certain conditions.

Feature selection methods mostly employ filter or wrapper subset evaluation functions (SEFs). Wrappers are basically classifiers, and can be some of the same ones that can converge to Bayes classifiers. Therefore, you can decouple the feature selection task and classifier choice task:

(1) Find many good feature subsets using a wrapper method that uses a Bayes-related classifier.

(2) Using the candidate subsets, evaluate Bayes-capable classifiers that match the number of classes and the amount of training data that you have. Or, try all of them.

Short answer is it is basically trial and error.

Trial and error, when executed correctly (using k-fold validation for example), is an optimal approach.

Feature selection must be done, as it is the step which makes the job of further classifier easy, it accelerates the classification, otherwise also we may input just all extracted features to classifier, and it will work, but now there is more burden on classifier, burden to memory,processing etc. Hence it depends on problem in hand and number of features extracted.

Its good to do feature selection, as it helps to get quality features which in turn improves the overall classification.