salem,

The book in the link bellow is about feature selection for high-dimentinal data. http://www.springer.com/us/book/9783319218571

I think the best way is to use filter based feature selection algorithms like : information gain, GRatio,....

It doesn't  seem correct if you do a feature selection for subsets of the features at a time. I mean if you selected at random the subset, you can discard some important feature from the begining.

Another solution is to use filter algorithms (since they are fast) to do a first selection. than apply another FS algorithm and so on.

Principal Component Analysis

Linear Discriminant Analysis

Auto Correlation Function

Principal Component Analysis , Linear Discriminant Analysi and some clustering method can be used to select best features from very high dimensional data. 

Thank you for your answers I am well aware of these techniques but my question was high dimension data that does not fit in memory. 

Maybe it will be better to solve this problem in a more 'hardware' way, instead of considering the altered algorithms for subsets. In my opinion, if the data are divided and handled respectively, its meaning may always be weakened no matter what algorithm is applied, since the correlations between subsets are cut down.

For your situation, the memory of a single computer does not fit in the data, a direct thought is to increase the memory by adopting a group of several computers. So maybe you can review some work on distributed computation to see if this can help you. 

One way to approach the problem could be distributing the data separating by features. In order to keep information about the correlation between variables, projection or random projection can be considered to be exchanged between nodes. One the communication is over, some embedded feature selection algorithm like lasso or elastic net can be executed. I do recomend some of this embedded approaches since there are some theoretical results that holds in the distrubted estimation. Take a look at this paper for reference https://arxiv.org/abs/1506.02554.

Ok. my problem is I have 10000 features and 3750 data points. The dataset is not sparse. If I use

- PCA : the eigen values are complex and few are negative.

- Correlation: most of the features are not correlated.

How to deal with this?

What kind of problems are you trying to solve? Regression, classification, others?

For classification, as a simple baseline, you might try to rank fetuses based on their mutual information w.r.t. the label and retain the top ones. Too account for interaction you can also you forward (or backward) selection.

Best,

Andrea

Not say directly.

Filter Methods:

Correlation Analysis: Calculate the correlation between features and the target variable. Select features with the highest correlation.

Variance Thresholding: Remove features with low variance, as they may not provide much information.

Wrapper Methods:

Recursive Feature Elimination (RFE): Start with all features and iteratively remove the least important ones based on model performance.

Forward Selection: Begin with an empty set of features and add one feature at a time based on model performance.

Backward Elimination: Start with all features and remove one at a time based on model performance.

Embedded Methods:

L1 Regularization (Lasso): Encourage sparse feature selection by adding an L1 penalty to the model's loss function.

Tree-Based Feature Selection: Decision trees and tree-based algorithms can rank features by their importance.

Dimensionality Reduction Techniques:

Principal Component Analysis (PCA): Transform high-dimensional data into a lower-dimensional space while retaining as much variance as possible.

Linear Discriminant Analysis (LDA): A supervised dimensionality reduction technique that maximizes class separability.

Feature Importance from Machine Learning Models:

Train a machine learning model (e.g., Random Forest, Gradient Boosting) and use the feature importance scores to select the most relevant features.

Regularization Techniques:

Use regularization techniques in machine learning models (e.g., Ridge or Elastic Net regression) to automatically perform feature selection by shrinking coefficients of unimportant features.

Univariate Feature Selection:

Apply statistical tests (e.g., chi-squared test, ANOVA) to select features that are significantly related to the target variable.

Sequential Feature Selection:

Implement algorithms like Sequential Forward Selection (SFS) and Sequential Backward Selection (SBS) to iteratively select the best subset of features.

Hybrid Approaches:

Combine multiple feature selection techniques to leverage their strengths and mitigate their weaknesses.

Cross-Validation:

Use cross-validation to evaluate the impact of feature selection on model performance. This helps in selecting the most appropriate subset of features.

Expert Knowledge:

Involve domain experts who can provide insights into which features are likely to be most relevant to the problem.

Regular Monitoring:

Continuously monitor the performance of your model with different feature subsets, especially as data evolves.