Feature selection per se is becoming less and less popular in the production teams that I work with. L1 regularized regression (Lasso method) is more and more popular.

Partial SVD should not be that expensive if you use something like the Tropp, Halko and Martinson sort of random projection algorithms. I have decomposed matrices with hundreds to thousands of features and hundreds of thousands of training examples in a few minutes using R with these methods. That said, I don't usually recommend SVD given the effectiveness of sparsity inducing regularization.

As another poster said, full on regularization is NP complete. This is comparable to L0 regularization which is plagued by multiple local minima, hence the difficulty. L1 regularization, on the other hand, is a convex problem if applied to logistic regression and thus is eminently feasible. Large ad targeting systems routinely use these methods to sovle problems with billions of features.

Hello,

Absolutely, this is the problem known as the "curse of dimension". So, keeping 1500 features not the learning time will be huge (example of neural networks) but also classification rate obtained will not be reliable and accurate. Therefore, it is obvious to proceed by an approach of reduction of dimension is inevitable and essential. By combining an approach to selecting the best attributes and a classification method, you will manage to get a smaller feature vector with wished classification rate.

Good search

NI

If you use a Decision Tree with pruning, the result model is not neccesarily use all features for the classification. With Perceptrons the features that does not adds knowlegde (with small variance, for example) are going to be associated with small coeficients. But in general is important do the feature selection after put the data in a Machine Learining Model.

There are several methods to do feature selection, try to search for example "Ridge", "Lasso" or "Principal Component Analysis"

Kind Regards

Ykk

You can do this with genetic algorithm, make each chromosome as binary with 1500 length (your features number), and the fitness function is the learning process with the output "error rate", then you can tell the GA to minimize the error of learning process by selecting some features or ignoring them, if the value of gene is 0 the counterpart feature will be ignored and vice versa if value of gene is 1 the counterpart feature will be selected,

Feature selection in this setting is np-hard i.e.: takes time to solve for 1500 features.

A genetic algorithm or other soft computing technique can give you an approximation of the best feature vector subset but doesn't guarantee to find the best one.

I suggest, you try to eliminate features that according to previous knowledge may be discarded. Then you could remove highly correlated features. Finally, you can try different feature selection methods like the GA described by Osama.

In many practical cases with a large number of features, a simple forward selection does a good job:

Wrapper approach:

1. For all features: Design separate classifiers with only one feature and select the feature with the best accuracy

2. For all non-selected features: Add an additional feature to the feature set and design a classifier and select the feature with the best accuracy

3. Stop if no relevant progress, otherwise continue with step 2

Filter approach:

Almost the same procedure, but use methods like Multivariate Analysis of Variances or Mutual information instead of classifier.

This approach did not guarantee the optimal solution, but often a reasonable one.

I would suggest vw-varinfo from Vowpal Wabbit package. Vowpal Wabbit contains very fast, efficient and scalable learning tools. vw-varinfo can be used for very large data sets (e.g. 200k features in text mining). Here you can find more useful information and examples:

https://github.com/JohnLangford/vowpal_wabbit/wiki/using-vw-varinfo

You would have an output like where RelScore is what you want. RelScores of around zero must be removed.

FeatureName HashVal MinVal MaxVal Weight RelScore

^bread 220390 0.00 2.00 +0.0984 55.36%

^icecream-sandwich 39873 0.00 1.00 +0.0951 53.44%

^snapple 129594 0.00 2.00 +0.0867 48.61%

^trailmix 215350 0.00 1.00 +0.0708 39.53%

^peanut 187714 0.00 8.00 +0.0464 25.49%

^pizza 32162 0.00 4.00 +0.0420 23.00%

Constant 116060 0.00 0.00 +0.0020 0.00%

^peas 126345 0.00 1.00 +0.0002 -1.01%

^quinoa 179283 0.00 1.00 +0.0002 -1.01%

^salmon 215140 0.00 2.00 -0.1015 -59.42%

^chicken 189058 0.00 1.00 -0.1133 -66.17%

^salad 171971 0.00 2.00 -0.1256 -73.24%

^mayo 187932 0.00 1.00 -0.1570 -91.28%

^oliveoil 69559 0.00 1.00 -0.1570 -91.28%

^egg 565 0.00 1.00 -0.1722 -100.00%

You can also try selective attention artificial neural networks (e.g. Sigma-if), which automatically select most important features for classification during training. What is interesting in this approach - as in most cases one reduced subset of features will not give optimal results for all data vectors, Sigma-if will give you features subsets tuned for each training vector. Then you can simply have subsets for each class in the trainig data. And the computation time on a sequential machine is smaller in comparison to ANNs without selective attention, as Sigma-if automatically uses (or not-uses) connections connected with features that are important (or not important) for the given data vector. Selective attention - as e.g. in humans - reduces computation cost and time while increasing classification accuracy.

@Maciej Huk, This sounds very interesting, how many classes are you usually looking at?

Information Gain and Brute force algorithm

Information gain does not help when features are correlated.

Unfortunately the problem is NP-hard so brute force may take too much time.

@Jens Allmer, During my research - two to 19 classes problems (average: 4 classes). Please ask me if you will need anything more in this subject.

We can decompose the data set with using many decomposing techniques like PCA , ICA(Independent Component Analysis) and SVD(Singular Value Decomposition) techniques, then apply for classification may get accurate results

I have good experience with Bipropagation. It enable fast and reliable learning with very high dimensional input space.

You can use minimum redundancy maximum relevance techniques....literature survey will guide.

@M. Murty

SVD , PCA or ICA are expensive approaches and are not suitable for this goal. Given a matrix of M (samples) x N (features) , a full SVD requires O(MN^2). Even k-rank SVD requires (kMN).

The mentioned selective attention Sigma-if neural network is trained with slightly modified error backpropagation algorithm and its training in worst case is O(K^3), where K is the number of weights (connections) within the network. So it is standard cost of training the perceptron neural network.

We frequently apply a learning vector quantization algorithm (LVQ) with the additional feature of relevance learning. Relevance learning weights the features (input dimensions) for optimal classification (GRLVQ). This approach we developed in 2002 and became very popular and successful. if you are interested in more details/assistence, please contact me

You can use a wrapper based technique for feature selection. According to my personal experience, genetic algorithm (GA) is a robust approach to find the most informative feature set among a huge number of features.

Feature selection per se is becoming less and less popular in the production teams that I work with. L1 regularized regression (Lasso method) is more and more popular.

Partial SVD should not be that expensive if you use something like the Tropp, Halko and Martinson sort of random projection algorithms. I have decomposed matrices with hundreds to thousands of features and hundreds of thousands of training examples in a few minutes using R with these methods. That said, I don't usually recommend SVD given the effectiveness of sparsity inducing regularization.

As another poster said, full on regularization is NP complete. This is comparable to L0 regularization which is plagued by multiple local minima, hence the difficulty. L1 regularization, on the other hand, is a convex problem if applied to logistic regression and thus is eminently feasible. Large ad targeting systems routinely use these methods to sovle problems with billions of features.

We wrote something on this which is still being cited. Includes math for exhaustive search, sequential forward and backward selection, plus L take away r algorithm, max-min feature selection. Please see (ask me if you want the pdf).

Biosignal pattern recognition and interpretation systems. 2. Methods for feature extraction and selection

Ciaccio, E.J. ; Dunn, S.M. ; Akay, M.

Engineering in Medicine and Biology Magazine, IEEE

Issue 4 • Dec. 1993

Here I attached it, didn't realize I could do that before. Scroll down toward the end to Feature Selection Methods.

In this paper, authors had 1,596 000 features. For strong features selection they used AdaBoost algorithm and it worked pretty well. Maybe you should check it out

Article Boosting Sex Identification Performance

In this paper, they refer to projecting *down* to several billion features. The way that this works at all is L1 regularization.

http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/41159.pdf

One interesting approache is using Genetic Programming as an automatic feature extraction technique. Through its evolutionary cycle, the more relevant features survive till the end while the weak ones are excluded and die. There are numerous references on the topic. The technique is very powerful. However the performance is an issue.

I recommend reading "A Survey on the Application of Genetic Programming to Classification" for PG Espejo, S Ventura and F Herrera.

LDA and PCA algorithms are useful for feature selection.

I think most modern ML approaches can handle 1500 dims, so long as they are all relevant to the problem. If you have junk in your vector then the only thing I know of (other than whats already been suggested) is a winnow style algorithm, although I have not tried the approach myself.

http://en.wikipedia.org/wiki/Winnow_(algorithm)

High dimensional low sample size data set is an objective of interest in most studies.

LDA and PCA are feature extraction not feature selection . Instead you can use methods developed for this purpose. fore example Elastic net is a combination of L1 and L2 penalties which is proposed by Hastie 2005. I recommend using Penalized package provided by Goeman for R.

As already proposed in previous answers, there are a lot of methods that you can try, depending also on what is the desirable outcome (dimensionality reduction or feature selection).

Assuming we only focus on feature selection, generic techniques (independent of the application domain) include:

- Stepwise (forward/backward) feature selection, e.g. http://www.cs.princeton.edu/courses/archive/fall08/cos436/Duda/FS/stepwise.htm

It is a very simple heuristic method, which works well for some cases but not always.

- Some kind of regularization. L2 methods (e.g., Ridge regression) are penalizing groups of coefficients (therefore you will not drop specific features). On the other hand, L1 methods (e.g., LASSO) penalize independent coefficients and can result in dropping some features and getting a sparse solution. Elastic Net (as indicated in a previous answear) combines these methods with different weights (typically selected by cross-validation).

- Feature selection incorporated in the learning algorithm itself: for example, Random Forests are an ensemble method based on decision trees as the template classifier. Each tree is trained with a subset of the training set (bagging) and a subset of the features and the result combines the outcome of the trees. It works very well in practice. Moreover, if your target is insgiht, the out of bag error can also be used to rank the features.

You should first prune the data for extract the important features; to do this, there are several methods, like: Principal Component Analysis, Hebbian Networks, LDA, etc.

PCA analysis is the best way to "thin" the data set where we can control the percentage of information loss by suitably controlling the number of independent features to be included in the analysis.

1.Feature selection,T-test Statistics

2. Discretization by setting up a cut-off.

You can use Principle Component Analysis to select most informative features from a big feature set.

High dimensional reduction is one of the important step for Data analysis.

Dimension Reduction categorized into two

1. Feature extraction

2.Feature selection

Feature selection :

Filtering model

wrapper

hybrid

Feature extraction :

For Feature selection there are many technique are in the literature. Most popular techniques are Principal component analysis(PCA) , Independent component analysis (ICA) , Singular Value Decomposition (SVD) , LDA(Latent Discriminent Analysis) etc

Using of feature selection method based on data set .

for example PCA is good for image data

SVD is good for text data

1500 is really huge no for selected features. One really needs to know the nature of application.

If it is image or pattern classification and image size is 40 x 40 total pixels come out to be 1600. then how can we have 1500 features? There must be some redundant features.

e.g. if image is size normalized, there is no point in taking length, width as feature. also area =length*width , hence area is redundant feature. In this way we can use human intuition to reduce the features. as these unnecessary increases computation time and e memory.

After above step, we can go for principal component analysis or SVD as said by M. Murthi.

Large no of features may create confusion in the classifier stage and reduce accuracy.

Selection of proper type of features and in correct quantity may be another area of research. One have to be judicious as per requirement of the research problem

dear Bilal,

Linear Discriminant Analysis as well as Principal Component Analysis work very well for your purposes, I suppose.

Which Feature selection algorithm works well for GLCM features

It depends on a couple of things. First, if you want to keep the original features, you should not use methods like PCA. These methods give a combination of features and the original features are lost.

If you want to keep the features there are some alternatives. Filtering methods are the most easy methods to use, since they work independent of the classifier. According the the score of the features, you can choose the most relevant ones.

If you want to add the effect of your classifier to the feature selection you can use wrapper or embedded methods. These methods have higher computational complexity compared to the filter methods.

You can go further even with deep learning methods but it all depends on the amount and complexity of your data.

I suggest to start with simple ones, and then if necessary go for more complicated methods.

1. First of all, it is needed to check whether you need to re-sample the data-set in the case of having rare event problem by finding the distribution of classes.

2. Second, the features with more than 70% missing values can be removed. Note that this step is optional and in some cases it is better to keep features even with more than 70% missing values.

3. Third, consider a threshold and drop features in which their occurrence in a class is less than the defined threshold.

* Keep in mind it is not feasible to identify correlation through correlation graph. So feature reduction based on correlation can be a challenge.

4. Fourth, the random forest can be used for feature reduction based on feature importance having finished the fitting procedure.

5. Fifth, it is possible to apply L1 or L2 regularization through “Logistic Regression”.

Hello

Given that I do not know the details of your project and the necessary comments are almost long , I suggest you to read the below paper. I think this article may be useful to you.

Article A new approach for rotation-invariant and noise-resistant te...

In this paper, one of the main challenges is feature selection for classification of a large number of images in several classes.

I recommend to try linear discriminate analysis (LDA) , PCA or other machine learning algorithms such as decision tree which can intuitively identify most informative features depending on such technique

Build aprogram or use weka or orange application to extract the association rule between you interested class attribute and all other attributes

i think this is exactly a data mining problem and you can use its techniques.

Usually filters are recommended in the presence of such amount of features. I would try FCBF algorithm which is fast and additionally it deals with redundancy. Note however that poorer results can be obtained if there are small number of objects (cases, examples) in your data.

Other way can be logistic regression with regularization, lasso or elastic net.

If the sample you have is quite large (say few hundreds cases) random forest is also a good choice. This algorithm performs feature selection automatically and gives feature importance ranking.

It depends on your data type. IF your data is textual, there are a lot of feature selection algorithms such as Balanced Accuracy measure, BNS, Normalised difference measure. you can also read following paper: Comparison of feature selection methods in text classification on highly skewed datasets.

i suggest to see the following paper@Bilal Esmael

There are various tradation ML methdos to extractmost informative features among the feature set.

a. Scale-invariant feature transform (SIFT)

b. Linear Discriminant Analysis (LDA)

c. Principal Component Analysis (PCA) etc,

but better to use deep learning methods, in which you are not supposed to extract/select handcrafted features, beaause DL methods automatically extract the relevant features.