Conceptually, in feature selection there are two alternatives, feature ranking and subset selection. Feature ranking in turn, exists in two forms: wrappers and filters. Wrappers are cost-prohibitive and classifier-specific. Filters are efficient and classifier-independent, they estimate discrimination of each feature in an independent fashion. The classifier then picks features with high(est) scores to use in training/testing and predictions.
Now, the fact that filter only works on one feature at a time seems troublesome. Classifier uses selected features in a group; filter on the other hand never takes into consideration more than one feature ignoring remaining features.
( For two-class problems one feature might be sufficient, and so the problem only exist for a multi-class scenario. )
Thus, with filter concept (one feature at a time) and implementation (classifier contains several features) are in disconnect. And worse, with filter there is no protection against selecting highly correlated, redundant features.
In contrast, many subset selection techniques are considerate of features' interference. First, in information theory there are approaches working directly with mutual information and thus, obstracting redundancy. Another group of methods compute projection of data into either orthogonal space (like PCA, LDA, etc.) or to some manifold beneficial for data discrimination (including linear embedding and other techniques); this way they address feature redundancy as well.
So, with pitfall of redundancy in mind: Shouldn't we constraint feature ranking to subset selection exclusively while abandoning filters altogether?
OK (taking the risk that my head will be chopped off) as a layman I would say: NO.
First of all, banning never is a good idea!
Secondly, you name two advantages of filters : " Filters are efficient and classifier-independent". As a disadvantage you state : "with filter there is no protection against selecting highly correlated, redundant features". It seems to me the latter is a task for the classifier or some other chuck of code that has access to the meta-data. I mean, redundancy has to be kicked out at a higher level.
I don't know how, so: how do you detect redundant features when the number of features is very large?
Ranking in general have been considered pre-processing step, upstream to entering the classifier. This should be more true for filters (while in subspace construction as in wrappers the classifier gets involved). At this pre-processing step filters have access to the training data and their (training) ground truth as well.
So, filter should not be concern with the downstream classifier. There may be some confusion in terms: like LDA as a projection/pre-processing step has been long called a linear classifier itself. Again, I only question 'pure pre-processing' filter types -- shouldn't they be abandoned in favor of subsets on the grounds of their poor redundancy control?
As of challenges with large feature numbers: well, there is a cost to high-D feature space; everything is more difficult when features are plentiful. Correlation becomes ill-defined, it's more difficult to orthogonalize in high-D, etc. We assume that the number of features always finite, but computational challenges would rise, for some methods more than for others.
(Hmmm, why don't you get more responses.)
I suspect filters are used because of computational efficiency and that the drawbacks are taken for granted.
I don't know about redundancy detection (=control), but using filters can degrade the predictive power of the model.
An example; for convenience sake a low dimensional one.
Say I have three features and want a model to predict whether a tumor is benign or malignant. I use PCA to reduce the feature space by one dimension. Assume that the three features when projected on a plane still retain a good portion of their variance, then I would just proceed...
It might be that 'all' examples of the training set that WERE on one side of the plane, are benign and on the other side malignant. By using PCA without considering the classification, I then just threw away a very important dimension.
Is this an illustration of why you propose to ban filters?
First of all, considering filters as being independent of a classifier is arguable,
if not a (common) misconception. Consider, for instance, a correlation based filter or even more specifically a t-test. Such filter assesses to which extent the class conditional means are separated along a specific dimension. The more the better and if so, the corresponding dimension is believed more important.
But such a methodology is actually equivalent to a very simple but yet a true classifier: a univariate gaussian model to discriminate between classes.
In other words, use the accuracy of this predictive model to rank your features individually and, by design , you will get the same feature ranking as provided by a t-test.
The same argument is applicable to at least all filters which take into account the class labels when ranking dimensions (e.g. relief is a filter related to the kNN classifier). So, the true question should be whether you are ready to consider first a very simple classifier to select features (to benefit from its computational efficiency)
and then build a more complex (and preferably related) classifier for your eventual prediction task.
Secondly, while most filters are indeed univariate some do model feature dependence and redundancy. MRmR is probably a prominent example. By the way, this technique relies on mutual information which needs to be estimated and hence often relies on a Gaussian assumption (back to point 1) or the so-called Kraskov estimator using nearest neighbor techniques (back to point 1, again.)
Thirdly, beyond wrappers and basic filters, embedded approaches should be mentioned: construct a classifier and rank the features according to their respective influence in the classifier *decision* (absolute weight values in a linear model is a prominent example).
By doing so, you often get a better computational efficiency than pure wrappers
(exploring many combinations of feature subsets) or even some costly filters.
To get to the point: should filters be banned?
No, because banning is indeed bad ;)
No, because in practice they may be very hard to beat.
A very common example is feature selection from microarray gene expression data.
While it is commonly accepted that genes are largely dependent on each others,
those dependences are only partially known and difficult to estimate reliably when
you have 50,000+ dimensions and only 50 samples (a common pre-clinical study setting).
So, even though any univariate filter should be bad because it plainly ignores feature dependence, it might actually be quite effective relative to a more sophisticated technique lacking enough data to offer a reliable estimation.
Well, good post; I appreciate the effort and contribution.
"First of all, considering filters as being independent of a classifier is arguable, if not a (common) misconception. Consider, for instance, a correlation based filter or even more specifically a t-test. Such filter assesses to which extent the class conditional means are separated along a specific dimension. The more the better and if so, the corresponding dimension is believed more important.
But such a methodology is actually equivalent to a very simple but yet a true classifier: a univariate gaussian model to discriminate between classes.
In other words, use the accuracy of this predictive model to rank your features individually and, by design , you will get the same feature ranking as provided by a t-test."
Yes, you could call filters classifiers (and they might or might not be followed by another echelon of classifiers)... But this is off my point; and I want to stay focused on one point only.
I wonder about efficacy of those univariant estimates that don't consider influence of other features.
In my question I meant ONLY situations when the feature estimate is isolated from other dimensions (other features).
An example: correlation (Pearson or else) coefficient computed on sample labels and feature values for same samples would rank this feature (regardless of all other features). Computing rank for the next feature disregards the previous rank, etc. Thus, consequent application of this estimate does not result in a set of independent (or even orthogonal) features.
And this is exactly my concern about this (univariant) way. This defect in design does not prevent situations when one ends up with all highly ranked features that highly correlate, while those independent all have much lower individual ranks. I see this a fundamental concern.
"The same argument is applicable to at least all filters which take into account the class labels when ranking dimensions (e.g. relief is a filter related to the kNN classifier). So, the true question should be whether you are ready to consider first a very simple classifier to select features (to benefit from its computational efficiency)
and then build a more complex (and preferably related) classifier for your eventual prediction task."
No… I actually don't see how the 'true question' adds (or related) to my point…
"Secondly, while most filters are indeed univariate some do model feature dependence and redundancy. MRmR is probably a prominent example.
By the way, this technique relies on mutual information which needs to be estimated and hence often relies on a Gaussian assumption (back to point 1) or the so-called Kraskov estimator using nearest neighbor techniques (back to point 1, again.)"
Yes, yes… Preceeding Peng's work there were textbooks [particularly Haykin's "Learning networks ..."] full of textbook approaches (pardon tautology) to work around correlating dimensions... Some approaches do rely on Gaussian assumption (is that really a deal-breaker?).
But again, this is not the point.
"To get to the point: should filters be banned?
No, because banning is indeed bad ;)
No, because in practice they may be very hard to beat."
I understand, they might be a cheapest option, but still.
Subsequent application of the univariate filter (the way described above) does not garantee a set of independent features, w/o cross-talk. I thought that is enough to start worrying about flagging this univariate filter approach. I believe this may save legwork for some.
Well, thanks again for the contribution.
Using Particle Swarm Optimisation it is possible to select multiple features simultaneously in a filter-based approach. Results on a range of test problems show that this is a promising approach and applicable to a wide range of downstream classification techniques:
Bing Xue, Liam Cervante, Lin Shang, Will N. Browne, Mengjie Zhang: Multi-objective Evolutionary Algorithms for filter Based Feature Selection in Classification. International Journal on Artificial Intelligence Tools 22(4) (2013)
Further, the results are often competitive with wrapper based approaches.
Hope this helps.
Per my understanding, PSO does not estimate each feature in isolation from other features (as particle position/velocity in a swarm is affected by other particles' positions and velocities). At the same time, Particle Swarm optimization is one curious brand I have little experience with. I am interested and would like to take this off stage; here is my address: [email protected]. Thanks.
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Artificial Intelligence