Hi Daniel,

Even to my surprise, my first recommendation would be to try linear PCA (Principal Component Analysis).

PCA will yield a new vector base onto which you can project your data, each new dimension being a linear combination of your original features. As a side benefit, you will get to see how much each of your features contributes to each principal component. Even more, you will get to see how your features group, given their contribution to each individual principal component. After projecting to your PC space, you may remove the lesser components and use a subset of your [projected] features, or you can hand-drop those of your original features that do not contribute to your principal components.

Hope that helps.

Cheers,

Fernando

Thank you for your input, Fernando!

We have considered using data reduction approaches like (PCA) analysis to reduce input features. However, because we are attempting to classify Autism-Spectrum samples from Typically Developing controls (Autism is thought to involve pathophysiologic molecular heterogeneity; I.e. different genes causing same phenotype across different subjects, we've been hesitant to collapse our genes into principle components. I would love to hear your thoughts with respect to this nuanced information. Also would love additional suggestions from other SVM users.

Hi Daniel,

Let me see if I understand your problem: you have a multi-dimensional set of data from which you want to find an optimal subset of features for your classification method. So it is not simply dimension reduction for performance reasons, but rather you want to find out the specific contribution (for some undefined measure of "contribution") of each individual feature, and then select a subset of those of some interesting size. Furthermore, you suspect that available feature selection methods don't fully reflect performance in classification space, so your final feature subset is suboptimal, given your classification method.

My proposing PCA was not so much about using the principal components in classification (which might be an idea) but about the insights that PCA can give you about how features contribute to the overall variability of your dataset.

Most PCA tools (princomp in matlab, for instance) will yield several pieces of data as their result. One of them is the actual matrix that is used to transform from your original data to the new principal components space. That is, for any given original vector 'e = (e1, e2, ..., en)', you will have a new vector 'v = (v1, v2, ..., vn)' given by the relationship 'v = Me', such that 'v1' is the new vector's coordinate along the first princpal component, 'v2' its coordinate along the second and so on. What is interesting here is that, if you expand 'v = Me', you will get:

v1 = a1*e1 + a2*e2 + ... + an*en

v2 = b1*e1 + b2*e2 + ... + bn*en

...

vn = k1*e1 + k2*e2 + ... + kn*en

That means, the principal component PC1=(a1, a2, ..., an), PC2=(b1, b2, ..., bn) and so on. So, inspecting matrix M, you will see how much each one of the elements of your original vector, i.e., your original features, contribute to each principal component. So, if, for instance, (a1=0.9, a2=0.7, a3=0.0001, ..., an=0), you can deduce that your features 1 and 2 are the ones that contribute most to the first principal component and, in turn, to the whole data set. If you get (b1=0.03, b2=-0.04, ..., bn=0.00007), you can deduce that your features 1 and 2 are the ones that contribute most to the second principal component, but their contribution is opposed to each other.

Finally, your PCA extraction function should yield a weight vector, meaning how much each of your PCs explains about your data. So my approach would be to take the first PCs that add up to explaining say 80% or 90% of the data (which will probably be only your first four or five PCs), and study M to reveal which features contribute to those and how they interact.

The only drawback is that this method is agnostic about your classification method, and that PCA is linear. Nevertheless it is surprisingly powerful.

As a second thought, one of the problems associated with SVMs and neural networks is that they are relatively obscure in terms of revealing their models. With SVMs this is less of a problem, in my opinion, but it is still not obvious how to extract information about their models.

SVMs rely on 'support vectors', which are the vectors that lie at the border of your class data. Once you have trained an SVM on a particular dataset, its model will be precisely the set of support vectors that it has chosen to separate between classes. You may inspect those to determine which features contribute to them. Again, PCA might be an option, but this time AFTER your training (thus taking in consideration your classification method).

And finally, I may suggest using different classification algorithms such as decission trees or random forests, even if only to gain knowledge. These latter perform relatively well, are very fast to train, and they can explain how a particular feature helps classify your data.

Hope that helps.

Cheers,

Fernando

Hi, maybe you can also try with random forest. It is able to give you a feature importance score list together with the estimation of classification. For example there is the scikit package able to do this. I've found it faster than time consuming Forward Selection or Backward elimination techniques....

http://scikit-learn.org/stable/

bharathi sir can you provide a small demo in how to implement those feature selection methods on customer review dataset? becz i try this using python programming language along with nlp in my research.

Hi,

Feature selection is mainly divided into two groups: filter and wrapping.

The filter methods do not consider the relationship between the features and the outcome, but only analyze the variance and/or importance of each feature alone. In addition, filter mothods assess the correlation between all the possible pairs of features and from a highly correlated pair of features one can be omitted and one can be maintained.

On the other hand, the wrapping methods are based on the relationship between the features and the outcome. And an auxilary function is used to assess this relationship. So if you are choosing your most related features to be then used in a linear model, you need to choose them with a linear auxilary function (method). In the same way, if it is a non linear tree classificaiton problem you need to use the same method as the wrapping function.

The support vector machone with linear kernel does the feature selection step automatically, while the other two (polynomial and RBF) do not have this step implemetned in their algorithm. As far as I know, a linear method cannot be used as a feature selection method to be applied afterwards in a SVM with polynomial or RBF kernels.

As others suggested, PCA could be a solution. But if the significance of features used in your model is important to you (as for most of the life sience problems it is) then other algorithms may bring you much better solutions, such as Random Forest.