Dear friends, I got 80 independent variables and one dependent variable with 1000 datasets. I was performing stepwise regression with quadratic fitting. MATLAB memory is showing an error of "Out of Memory" in very starting of its execution. Definitely it is because of large number of variables. Now I am planning to select important (significant) variables first (out of 80 variables) to run my program. Could you please help me how to execute this task.
Paul makes a very good suggestion but that will only work for linear relations. You mention quadratic fitting so I assume you also have terms like x^2. BTW I would forget about stepwise. See Austin and Tu (2004), Automated Variable Selection Methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality.J. Clinical Epidemiology 57,1138-1146 to find out why stepwise is a bad idea. Either lasso or elastic net is much better. See Lander, J., R for Everyone, Chapt. 19. for a quick introduction. You can use something like this for your final variable selection. There are many packages available for these things. You also mention 1000 data sets. I don't understand this. Best wishes.
Dear Prof David Eugene Booth,
Thank You very much for your valuable reply. Today I tried using LASSO and Elastic Net. I have got the significant parameters from Elastic Net. My concern is how to use these parameters further in order to build the model. Weather
1. I should rely on coefficients from Elastic Net (though no interaction, no adjusted R2 value) OR
2. I should use these terms in stepwise regression (involves interaction terms, adjusted R2 value, but bulky model)
My motto is just to generate a equation along with high R2 value. Please enlighten me with your knowledge as I am new comer in the field of statistics.
In a similar data-set size, but with over 1,000 independent variables, I've been able to utilize step-wise regression via SPSS. Two key potential problems are: N/As or non-numeric data, or perhaps a IV(s) is extremely similar to the DV. Running a correlation matrix of the data could help reveal potential problem variables. Also consider transforming the data (ie LN, X2, 1/x), and utilizing VIF or TOL thresholds to reduce multi-collinearity.
Stepwise regression sounds good, but often doesn't work very well in practice. I can easily create examples where you'll never get the optimum result by taking a stepwise path. You must often discard a good result and even a better result in order to arrive at the best result. The problem you're describing is fairly small and it would be a simple matter to try lots of possibilities. The only reason this problem seems big is because you're using MATLAB. I work with data sets all the time that are over 100 MB and have hundreds of variables and tens of thousands of values for each one. I can process such data in seconds on a laptop that's over ten years old. Your problem is MATLAB. Get one of the many free C compilers, a good text editor, and chuck MATLAB. You can find any algorithm you need, like simultaneous equations, linear constraints, nonlinear minimization, etc. free on the web. Check out The Code Project.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression