Hi,
Till now I used SPSS but now I started using R and currently I am trying to run ridge regression.
In my data I have five IVs. two of them are actually covariates (age and sex). The other three are continuous x, y and z. When I checked the data I found that for x the VIF values with y and z >10. Therefore I decided to use ridge regression.
The problem is I don't really understand how to do it in R and there is no good video tutorial that I could find (nor do documents were very helpful with that). I don't know where in the code I should specify my data, how to specify my five IVs and how to specify my dependent variable.
Would be grateful for any help. Take in mind that I'm new to that.
I assume you already discovered the lm.ridge function in the MASS package. There's also a glmnet function in the glmnet package, which seems a little more complete.
Obviously, R can be difficult to use when starting out.
I'm really not familiar with these functions, so I won't be much help on the interpretation or even if they give you all the information you need, but...
In case you're real new with R, to use the lm.ridge function:
1) you have to install the MASS package. (Or maybe that comes pre-installed with R at this point.)
2) library(MASS)
3) ?lm.ridge
### This tells you about the function.
4) The example in that documentation is little tough to understand, but you should be able to run the example, and get the output and the plot.
The model specification is standard:
lm.ridge( y ~ x1 + x2 + x3, Data)
where y is DV,
x1 etc. are IV,
Data is name of the data frame your data is in.
(The example uses an abbreviated form, y ~ . for the model)
Note that in the plot and select commands that lambda lists the ridge coefficients. In the example a sequence of values from 0 to 0.1 by 0.0001 is used. This is actually specified inside the lm.ridge function.
Thanks a lot! Just wonder if you can take a look at the numbers (attached in image) and tell me if you think that ZElgenver is stable? I know that ZFlowgenver is unstable because it flipped sign, but ZElgenver doesn't flip sign, just change magnitude relatively fast compared with other stable variables if we look only on the ten first lambda changes.. (P.S. I arbitrarily selected lambda change by 1, didn't take any estimation of desirable lambda change..I understand now that cv.glmnet do that? Didn't figure how to work with it yet..)
I really have no idea how to determine if a variable is stable in this. I am so unfamiliar with ridge regression... It might make for a new question.
You might look at the last few slides here:
http://web.as.uky.edu/statistics/users/pbreheny/764-F11/notes/9-1.pdf
They have AIC, BIC, and GCV statistics, which I guess determine the best lambda???
cv.glmnet. Though as I'm new to R, it cause me many problems, still didn't find way to feed the data in properly.
Hi,
Concerning the question of stability, it is, in my opinion, due to the fact that green and red variables are correlated, highly correlated I would even say. A regression ridge does the following : it attributes penalties to the least informative principal components. So basically, the coefficients of the variables which are highly correlated with the least informative principal components are the first ones to be reduced when one increases lambda (cost parameter). Note that, as lamdda goes to infinity, all coefficients eventually fall toward 0.
As for codes about applications of the ridge regression, please have a look at the following link http://people.stat.sfu.ca/~cao/Teaching/852/Ridge.R .
Besides, as I perfectly understand why ridge regression is widely used (close formula for beta-ridge-estimator), if you want to operate variable selection, I suggest you tu use Lasso-regression by Robert Tibshirani instead.
Finally, note that both ridge and lasso regressions introduce biases into coefficient estimation process, so they might be complicated to use in econometrics. However, while I was browsing the internet to substantiate my prior knowledge (I know that sounds very Bayesian :p ) on the subject, I found an article that could be interesting to you :
http://faculty.chicagobooth.edu/christian.hansen/research/Regularized_JIVESubmit.pdf .
If you are interested by advanced regression techiques and more generally by machine learning algorithms and other cutting-edge methods, I also suggest you to use the Elements of Statistical Learning https://web.stanford.edu/~hastie/local.ftp/Springer/OLD/ESLII_print4.pdf which is a Bible, at least mine when it comes to statistics, buy you have probably heard about this latter one already !
Hope this helps a little,
Best regards !
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Statistics
- Statistical Software
- SPSS