Since there was a brief hint of SVD, I'd suggest you consider that independently of PCA. SVD can be used directly on a table of records versus attributes, rather than doing the covariance thing that PCA does. This is the basic approach of latent semantic analysis (LSA, see the papers by Landauer, Dumais, Deerwester, et al. The advantage is that using SVD on the raw record vs attribute data means that the singular values are actually very easy to interpret, as they are directly related to both the records and the attributes. Typical LSA will normalize the attributes just slightly, maybe using log counts rather than integer counts, and it's still fairly easy to interpret.

I believe that your dataset is quite small and PCA would be quite easy to use. However, you can basic clustering (k-means, k-medoids, etc) to generate a small number of clusters and discretize your data this way. Is there a particular reason you want to reduce the dimensions without losing much information? I believe this is the point of reducing dimensionality. You can set up PCA or SVD with a 90-95% variance threshold and you should keep more than enough information. Hope this works!

try AdaBoost, it will give you most discriminant dimensions.

Since there was a brief hint of SVD, I'd suggest you consider that independently of PCA. SVD can be used directly on a table of records versus attributes, rather than doing the covariance thing that PCA does. This is the basic approach of latent semantic analysis (LSA, see the papers by Landauer, Dumais, Deerwester, et al. The advantage is that using SVD on the raw record vs attribute data means that the singular values are actually very easy to interpret, as they are directly related to both the records and the attributes. Typical LSA will normalize the attributes just slightly, maybe using log counts rather than integer counts, and it's still fairly easy to interpret.

If your data set has a "class" or "label" attribute, classical rough set theory combined with bio-inspired algorithms is a great method for performing feature selection (check the paper "Rough Sets and Evolutionary Computation to Solve the Feature Selection Problem").

If you don't have a class attribute, unsupervised fuzzy-rough feature selection has been recently proposed with very encouraging results (check the paper below).

Article Unsupervised fuzzy-rough set-based dimensionality reduction

Have you considered least-angle regression?

.

PCA is indeed often hard to interpret because each variable may contribute to each factor and, moreover, such contributions can be positive or negative.

depending on your application (and in addition to trucated svd already proposed above), you might consider :

- sparse pca (pca with a sparsity constraint so that few variables only can contribute to each factor ; greatly simplifies the interpretation but contributions can still be positive or negative which is sometimes a real pain)

http://en.wikipedia.org/wiki/Sparse_PCA

- non negative matrix factorization (also reminiscent of PCA but this time the contributions of the variables to the factors must be positive, which simplifies the interpretation in many applications)

http://en.wikipedia.org/wiki/Non-negative_matrix_factorization

- sparse non negative matrix factorization (aims at the best of the both above techniques) ; have a look at

http://mmds.stanford.edu/slides2008/jin.pdf‎

which gives an example for comparison of the interpretation with svd or (sparse) nnmf

at the end of the day, a "good" technique for dimension reduction will depend on your application ; it sometimes makes sense to avoid negative contributions of variables to the factors, sometimes it does not, for instance.

.

As mentioned before, for high-dimensional datasets (dim > 10), K-Means clustering can be performed beforehand in order to extract most representative data and quantize them based on these.

Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), or Canonical Correlation Analysis (CCA) techniques can follow K-Means quantization step in order to extract feature vectors with reduced-dimensional size.

Sample it?

Personally, I like Hall's CFS method:

http://www.cs.waikato.ac.nz/~mhall/thesis.pdf

In the formulation of this method the space of possible feature subsets is explored. Subsets are given a merit where dependences between features and the class are prized, whereas dependences between selected features are penalized, being those dependences measured by means of SUS between pairs of variables. This formulation also provides a high degree of interpretability for experts. It is implemented in the free-software named WEKA:

http://www.cs.waikato.ac.nz/ml/weka/

All the best

You may also try the built-in feature selection methods of Weka Software in order to indicate the most informative features as well as the most redundant.

You can use Entropy in Decision trees (i.e. ID3) to keep only the good attributes and this task depends on the DM functioality.

Feature selection is useful, but you should consider some aspects. For example, despite the fact that PCA may be difficult to interpret, it can provide you the information about the intrinsic dimension of the data. This can be found from the exploration of the normalized (by their trace) eigenvalues sorted in descendent order, where the intrinsic dimension is approximately given by the number of eigenvalues that sum 0.95 (they contain 95 % of the variance). Feature selection should be used taking into account the relations or interactions among variables, e.g. by using a genetic algorithm and taking care that the number of selected features is similar to the effective intrinsic dimension. Furthermore, if the final goal is classification you should consider the possibility of nonlinear feature extraction like the one that is performed by nonlinear kernels. The visual data exploration in 2 or 3D by a PCA can give you insights about the linear or nonlinear distribution of your data. The "No Free Lunch Theorem" applies also to feature selection, there is no best method in general, it depends on your data.

You can find some at http://www.abonyilab.com/classification

Try to use ID3 if your purpose is Classification and find the attributes that do not contribute to the classification process and delete them and build the classification tree after deletion. Try to use Pearson product-moment correlation to find the similarity between attributes and which can be deleted.

In a recent paper combining seismic and gas emission data we had to shrink the dimension of the seismic part.Using PCA did a good job. Another issue is the use of clustering...by the way, we got quite promising results with Kohonen Maps....which concentrates the data set in a limited number of prototypes. At the same time it is often possible to project the prototypes lo a low dimensional representation space (the "map"). You mght see our paper by Messina & Langer, 2011, Computer and Geosience (CAGEO) where we published a computer code "KKAnalysis" -it can be downloaded for free...

According to internal testing done in our lab, ISOMAP is the best among 10 different dimension reduction methods that are available in R

.

@Nicolay

is it possible to share a report or a link for this internal testing ?

.

I am sorry but this is an unpublished data yet... I will share it once it is out. But the essence of the test was: we profiled immune cell populations of normal and cancer patients. Each patient was profiled in 4 technical replicates and for each replicate we computed the counts of different cell types and then ran dimension reduction to get the patients distribute on a 2D plane. Our requirements for a good-quality 2D map were:

1) all the normal patients should be grouped close together

2) the technical replicates of each patient should fall closely together

Under those criteria, ISOMAP consistently gave us the best result

You could select the relevant subset of features using one of the methods described and implemented:

http://sfb876.tu-dortmund.de/auto?self=$ddwpqancao

If you does n't want PCA then you can go for various attribute selection algorithms like InfoGain or GainRatio.

Dear farideh,

you clearly wrote that "I don't want to do something like PCA, because the interpretation of the model built on them would be hard". This means that you need some technique to reduce data dimensionality, but maintaining the observed variables (with their relative meaning) as they are during the recording process. Besides the technique described by the colleague Fabrice Clerot from Orange Labs, lastly you can think to reduce each variable dimensionality by applying some simple, but effective operation (such, for example, instead of considering all the M samples, to take only N parameters extracted from the variable itself by averaging the samples contained in all M/N blocks of data)

L1 PCA is one of the techniques that uses multiple linear regression and SVD iteratively to reduce the dimensions. The interesting thing here is that the dimensions which are eliminated in each iteration can be interpreted easily and at the end of the iteration the first principal component is obtained. Thus the interpretation of which dimension is the principal one can be easily done.

I would suggest to use self organizing map for dimension reduction and also for the cluster identification for ur problem

t-SNE is an awesome tool for generating a low-dimensional representation that preserves the local distribution features. I have a Java implementation, if you are interested, and Matlab, R, C and Python are available online

http://homepage.tudelft.nl/19j49/t-SNE.html

So, there are various ways to do feature selection, although most do not give optimal solution. For example, you can try SFS (sequential forward selection), SBS. If these methods were not useful, then after that I would suggest using GA (genetic algorithm)

Why do you want to reduce the dimensionality to begin with? Are you doing classification on this data, or do you want to visualize it, or ?