I am doing feature subset selection from a set of fifteen features. Presently, I am using principle component analysis (PCA). PCA is actually generating a set of new features, each is a linear transformation from the original elements. I have also performed feature selection using Kernal-PCA, but I didn't get good results. Are there other methods for feature selection in an unsupervised manner (i.e. I don't have training data set)?
I have recently done a review (Book Chapter) about the same topic.. Feature Selection for Clustering: A Review
http://www.public.asu.edu/~salelyan/MyPub/FS4clustering_chapter.pdf
I totally agree with William Caicedo when he said PCA is NOT a feature selection method.. and that we cannot say "best" in the field of data mining and machine learning.. It is data dependent.. There are a few methods that may work reasonably good but we cant say they are the best..
I would suggest to use SPEC ( Spectral Feature Selection) it is both supervised and unsupervised. You may find a Matlab implementation of SPEC in our Feature Selection Package. http://featureselection.asu.edu/software.php
I would like to tell you first that PCA is not a feature selection technique. PCA is a dimensionality reduction technique, and the features it produces often are not readily interpretable. In feature selection you have the wrapper approach (which seems to be the one you need to avoid), and the filter approach. Filter methods use some metrics in order to establish which features carry more "weight" in the data set. Also you could try with clustering techniques such as Projective Adaptive Resonance Theory or Kohonen Self-Organizing Maps in order to identify the most prominent dimensions that explain your data variability. Also, you could consider doing a Variance Analysis (ANOVA) for pretty much the same goal.
It is difficult to say which is the "best", since they all have their own weaknesses. You'd be better off trying several techniques to find the best performer in your specific case.
I agree partially. I applied a non-linear algorith called curviliner component analysis which can do the same as PCA but with no limitation about linearity.
See
Sarcia’ S.A., Cantone G., and Basili, V.R., “Adopting Curvilinear Component Analysis to estimate Software Cost Estimation Accuracy,” British Computer Society, BCS-EASE08, Bari (Italy) 2008
or
Sarcia’ S.A., Cantone G., and Basili V.R., “Auto-associative Neural Networks to estimate Accuracy of Estimation Models”, in the book “Artificial Intelligence Applications for Improved Software Engineering Development: New Prospects (Eds: Farid Meziane and Sunil Valeda), IGI Global, USA.
Feature ranking and best feature subset using mutual information. Neural Computing & Applications. September 2004, Volume 13, Issue 3, pp 175-184
It is not possible to say which is the "best method" as it would depend on your data. In addition to the suggestions already given I can suggest you the following:
Pabitra Mitra, C. A. Murthy, Sankar K. Pal: Unsupervised Feature Selection Using Feature Similarity. IEEE Trans. Pattern Anal. Mach. Intell. 24(3): 301-312 (2002)
http://www.computer.org/csdl/trans/tp/2002/03/i0301-abs.html
the selection of feature set by description geometric or structural description from grammar concept or physic ariables.
Some of unsupersived feature selection techniques are based on:
1 - distance measures : the goal is to maximize the distance between classes (minkowski, euclidean ,..) if data normality is assumed you can use (mahalanobis, battacharya,...)
2- information measures: determine the information gain from a feature or a mutual information between features and a class label
H. Peng et al: Fs based on mutual inf: criterai of max-dependency, max relevance and min-redundancy, PAMI vol 27, n ° 8 pp 1226-1238, 2005
3. dependency measures: correlation measures between variables
You can find an overview on feature selection together with a plug-in to the RapidMiner system performing them:
http://sfb876.tu-dortmund.de/PublicPublicationFiles/lee_etal_2011a.pdf
Unsupervised feature selection needs to maximize an objective function as supervised methods optimize the fit to the class labels. Several such objective functions are built-in some of the methods. The report gives a nice overview and the system allows you to try it all out!
I have recently done a review (Book Chapter) about the same topic.. Feature Selection for Clustering: A Review
http://www.public.asu.edu/~salelyan/MyPub/FS4clustering_chapter.pdf
I totally agree with William Caicedo when he said PCA is NOT a feature selection method.. and that we cannot say "best" in the field of data mining and machine learning.. It is data dependent.. There are a few methods that may work reasonably good but we cant say they are the best..
I would suggest to use SPEC ( Spectral Feature Selection) it is both supervised and unsupervised. You may find a Matlab implementation of SPEC in our Feature Selection Package. http://featureselection.asu.edu/software.php
I think there isn't the best algorithm. You should find the best algorithm for your data. And about the PCA try normalization like Z-Score before using PCA.
what is best algorithm? spatial and temporal complexity . Pca reducing the dimension loosing accuracy.
All I would say is if you are trying to get rid of noise in your data via pruning unhelpful features, PCA wont help you in a meaningful fashion. If you need to reduce dimensionality and you think all features are needed then PCA is a good option. But Judging by your question context PCA is not what you want right now.
I personally would suggest trying a heuristic based approach on the total heterogeneity of the feature
https://it.mathworks.com/matlabcentral/fileexchange/56937-feature-selection-library
PCA tries to find maximum variance in the predictor variables, but this is not necessarily the best variables for prediction of the response variable. Partial least squares is an alternative and long-used method that is similar to doing PCA but it also considers the variables ability to predict the response variable. See 2013, Kuhn M., Johnson K., Applied Predicitive Modeling, 6.3 Partial Least Squares for an insightful description.
Here is a very good source of PCA including presentation and source code
Presentation Principal Component Analysis (PCA) : An Overview
Article Principal component analysis - a tutorial
Alaa
Principal Feature Analysis (PFA) Article Principal Feature Analysis: A Multivariate Feature Selection...
Sparse PCA (SPCA) https://en.wikipedia.org/wiki/Sparse_PCA
One way to go about to this would be to perform SVD and find a low rank approximation (D') that reduces the Frobenius Norm. Post which, |D - D'| should give you an outlier score of each entry in D (Reasoning being D' is a low noisy representation of D as it is a rank-k matrix where k < min{n,d} where n are the rows and d are the columns of your initial D). Thereafter, add the square of these scores for each column to get a predictor score for each predictor. Depending on how the scores are you could get rid of a few predictors and choose a subset.
- Badges
- Science topic