My data-set has 23377 instances for training (7792 for testing). The dimension of the data is approximately 28000. Each instance represents a document and the target variable is the age of the document's author. I want to perform a regression over the data with all the features (± 28000) and/or do some selection feature process to work with a regression method afterward.
What I was thinking to do is 1) using ridge-regression (with the ± 28000 features) and 2) perform a feature selection method to see the impact of dimension reduction with ridge-regression. Ridge-regression over the 28000 features is taking a very long time; it is still running in a server (I use Sklearn, Python). After that I tried with SVR (Support Vector Regression) which is taking a long time too. For feature selection I was using χ2 (CHI2) over the same data for a classification problem but for doing regression I have to use other method. The method I want to use is Lasso (http://goo.gl/2DzM0K).
Is there other regression method for this high dimensional problem? Or Can you recommend a method to do a feature selection?
I suggest ELM or on of its variants. https://www.researchgate.net/publication/40684804_OP-ELM_optimally_pruned_extreme_learning_machine?ev=prf_pub
Article OP-ELM: Optimally pruned extreme learning machine
I suggest ELM or on of its variants. https://www.researchgate.net/publication/40684804_OP-ELM_optimally_pruned_extreme_learning_machine?ev=prf_pub
Article OP-ELM: Optimally pruned extreme learning machine
You could try with Genetic Programming: there's a commercial tool called Eureqa that comes with a free licence for academics. Since the algorithm is stochastic, there is no guarantee that the solution it will find will be the best one, but it's worth a try :-)
http://www.nutonian.com/
You could try a fast Kernel Logistic Regression algorithm, http://www-ai.cs.uni-dortmund.de/SOFTWARE/MYKLR/index.html, or maybe an import vector machine, http://www.ipb.uni-bonn.de/ivm/, or lasso regression, http://statweb.stanford.edu/~tibs/lasso.html, if the data is sparse?
You can take a look to the tools developed at the Image and Signal Processing Group in Valencia, http://isp.uv.es/soft.htm. I believe some of the tools (written in Matlab) may fit to your problem.
I suggest to use WEKA software. In WEKA, you could try M5P algorithm.
I suggest to take matrix column approximation approach.
Currently, my group is developing sparse sufficient reduction methods that may serve your purpose. We have not submitted paper for publication. However, if you can share the data we can work together to solve your problem.
dear firnd
try to use logistic regrtession method.
see my publication for more details.
All the best.
Best regards
Dr.Indrajit Mandal
Hi Angel: Large scale regression problems can be considered using logistic regression method, however to deal with large scale data you could also try to incorporate l1-norm penalization (which induces sparsity, very useful in large data due to the sparse structure of the solutions). Have a look at the book: Optimization for machine learning by SRA, NOWOZIN and WRIGHT.
try using R packag "flare" which supports following high dimensional regression methods like
LAD Lasso, SQRT Lasso, `q Lasso, and Dantzig selector..even some more extensions are available....
Yes, I can. Next week I will finish a paper in which I will present a novel approach to this problem. I will send it to you. Please remind me.
Elastic Net may give a better result? But it should be slow too.
Given such a big data sample size and high dimension, some pre-processing such as feature selection, dimensionality reduction may be necessary.
All models suffering from "curse of dimensionality" are not suited for these kind of problems (SVR for example) with high dimensionality.
Try to use Multi Layer Perceptron (MLP) trained with Bayesian Regularization (BR) method.
MLP does not suffer from curse of dimensionality, and BR tends to get null weights in those input features that have no interest for the task, so you will be able to see if the task is possible (I think it is quite difficult) and which features are appropriate for it.
In addition to ANN recommendation, I suggest you to perform feature selection using gamma test.
Projection Pursuit Regression is also a good alternative (ppt R function)
I suggested functional linear model with quadratically regularized functional principal component analysis that are currently developed. You can contact with me about this new model for genomic data analysis.
Dear Angel
I suggested that this data should be reduced its dimensions based on the PCA or you can be applied the radial function to move data from real-space to feature(radial) space, please see radial basis M5Tree models proposed by me.
also you can used a nonlinear training approach using optimization algorithms to calibrate the data by any linear and nonlinear regressed functions.
Regards
Behrooz
Quadratically regularized functional principle component analysis or quadratically regularized functional canonical correlation analysis may have higher power to reduce the data dimension than PCA, particulaly, if data are a function of a parameter. Radial basis can be taken as a basis function in functional principle component expansion.
Dear Xiong
you can be applied the PCA-based RSM or radial basis NN.
Regards
Behrooz
I recommend that you first use the LASSO method for feature selection, and then use the ensemble algorithm such as random forest to do your regression
- Badges
- Science topic
- Similar topics
- Computer Science and Engineering
- Programming Languages
- Python