In my model, I experience a multicollinearity problem in least squares estimation. Therefore, I decided to use the ridge regression method. I examined the variance inflation factors (VIFs).
In the beginning, some of the VIFs for the variables were above 10 and the R-Squared statistic was 61.24. The value of the ridge parameter in my model is 0.1. Now, VIFs are around 2 for all the variables in my model. Finally, the R-Squared statistic indicates that my model as fitted explains 57.59% of the variability. I believe that for my model this R-Squared statistic is too low.
Dear Turkay, I'm afraid there's no better method to rise your R-sqr except that your regressors have latent dependency. If so, you can use path analysis to reach better results.
There are many regularisation approaches that you can use, Ridge regression is not the only one. You could, for example consider the Lasso (http://www.jstor.org/stable/2346178), Partial Least Squares regression (http://epubs.siam.org/doi/abs/10.1137/0905052) or Continuum regression (http://www.jstor.org/stable/2345437). Then there are many other approaches including kernel learning methods like Support vector machines regression.
Which is best? it all depends what you want to do. In general no one method is uniformly best. Some methods, like the lasso are designed to produce sparse models (not many predictors)_, that is they include a degree of variable selection. Other models like Partial least squares regression or continuum regression integrate over many variables. Which is more appropriate in you context? Do you want to identify a small number of predictors for future study or simply get the best calibration you can?
In high dimensional work the processes you go through to keep the results honest (i.e. cross validation, hold out independent validation sets, avoiding pre filtering - or filtering within the cross validation loop only) are very very much more important than which model you use. The higher the ratio of the number of potential predictors to the number of observations the more chance you have of finding spurious relationships. Above all you need to be rigorous with your validation.
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics