I am doing classification experiments using high dimensional feature vectors, which looks like a sparse vector , having a lot of uninformative and correlated features. I am using a random forest implementation with some modification, random selection of features is giving good results, but I was wondering if there is an easy way to get rid of redundant and uninformative features, which would improve the performance of the algorithm.
Redundancy and irrelevancy are very different concepts, having very different effects on the data. For a good intro to the curse of dimensionality, see "A survey on unsupervised outlier detection in high-dimensional numerical data" by Zimek, Schubert, and Kriegel (http://onlinelibrary.wiley.com/doi/10.1002/sam.11161/abstract and http://www.dbs.ifi.lmu.de/cms/Publications/OutlierHighDimensional). You may not be doing outlier detection, but the discussion of the curse of dimensionality is still applicable.
Having highly correlated attributes is not necessarily a bad thing. Indeed it may be a very good thing. See the papers on ultrametricity by Murtagh, et al. (http://bit.ly/17gNlyr) to understand why. The tl;dr version is that at high dimensionality, correlated features cause the contrast between intracluster point distances to shrink relative to intercluster distances, thereby enhancing the contrasts between classes.
I'm not saying you won't still want to do feature selection after reading what I've suggested, but at least you might have a better idea of which features you really want to select. Which method is best will likely be dependent on what your selection criteria end up being. Hope that helps. :)
Hi..
Feature Selection is basically selection of a subset of features from a larger pool of
available features. The goal is to select those that are rich in discriminatory information with respect to the classification problem at hand. This is a crucial step in the design of any classification system, as a poor choice of features drives the classifier to perform badly. Selecting highly informative features is an attempt
(a) to place classes in the feature space far apart from each other (large between-class distance) and
(b) to position the data points within each class close to each other (small within-class variance).
For more details: see chapter-4 of below book:
Book: An introduction to pattern recognition: A MATLAB approach
BY: Theodoridis, Sergios
You can also try WEKA data mining tool for feature selection, as in WEKA different feature selection algorithms are there.
Gud luck...!!!!
Sequential forward floating feature selection is easy to implement and it can give a good understanding of the best features after running several times over the dataset.
- Badges
- Science topic
- Similar topics
- Data Mining
- Classification