I have a data set with dependent variable (Y) which has three outcomes scaled 1 (bad), 2 (good), 3 (great). The data is variable-heavy and has collinearlity issue. Typically, in OLS regression that issue can be alleviated using either Ridge or Lasso regression. Can Ridge be used in Ordinal Logistic Regression? If yes, how? Thank you!
Yes, there are two R packages that may be of interest in this regard. Se the link:
https://www.google.com/search?q=r+ordinal+logistic+regression+lasso&rlz=1C1CHBF_enUS847US847&oq=r+ordinal+logistic+regression+lasso&aqs=chrome..69i57j69i64l3.20269j0j4&sourceid=chrome&ie=UTF-8
Hope this helps. Best wishes, David Booth
As David Eugene Booth reports, and very helpfully illustrates with an R link, yes you can.
But perhaps there is interest in probing a bit deeper. When you say that the issue of collinearity “can be alleviated,” you judge that this is indeed an “issue.” I would like to point out that it may not be. And, I would like to dig more deeply into the issue to specify why. Collinearity, or as is popularly used “multicollinearity,” has often been the springboard for ridge regression, and many other alternatives to OLS. I, along with colleagues, have done a good bit of simulation work with ridge regression and some other alternatives to OLS, although to this point, that published work has not included LASSO, but my expectation is that the same principles apply.
The first question is a mater of more precise definition. Is your interest in prediction or explanation? These goals are nicely defined by Kerlinger, and more specifically by Huberty. Prediction regards the degree to which the model generates predicted values for the criterion that are accurate. Explanation regards accuracy of estimates of the parameters – the population weights. We would usually think of both of these in a population sense; respectively, from a sample generated model, to what degree is accuracy obtained in predicting the criterion or estimating the weights in the population. It also may be that your goal is some combination thereof, perhaps with more interest in prediction than explanation, or vice-versa. At the extremes, my own opinion on this is that one can be interested in only prediction (my model, composed of various jockey, horse, and track variables wins at the track, and I have no interest in estimating the weights as parameters), but not only in explanation. In that latter case, it is hard to justify interest in the weight parameters if the model does not manifest some modicum of prediction accuracy.
Given that decision, the reason for the “may not be” above is that for the goal of prediction accuracy one will find that ridge regression (wherein one estimates k, or ks), and a variety of alternatives to OLS (rank reduction, from both a PC solution or common factor model, and naïve strategies such as equal-weighting), do not necessarily exceed, and can often under-perform OLS, and that this is not dependent on “collinearity.” An important determinant of these techniques’performance in respect to OLS is “validity concentration,” a notion and term cleverly coined by Richard Darlington. This is further complicated by the fact that some degree of collinearity is a necessary, but not sufficient, condition for validity concentration. So, in short, in just considering collinearity, one will find that these alternative techniques may underperform OLS or exceed OLS, dependent on validity concentration (and other sample characteristics, e.g., n, rho2), whilst OLS is insensitive to both collinearity and validity concentration. All that I say applies to both relative accuracy (r2.y,yhat) and absolute accuracy (MSEy,yhat), although the OLS vs. alternative operating characteristics render different “crossing points.”
We have done many simulation studies (in respect to predicting a continuous y, and “chotomous” y – classification) in respect to these issues. The first that manifested these results, although this was not the primary focus of the study was:
Article Ridge regression and some alternative weighting techniques: ...
Although there are several candidates, an example treating the question with more specificity might be:
Article Prediction, Explanation, Multicollinearity, and Validity Con...
And there are other examples is my “Multicollinearity, Validity Concentration and Prediction" Project at:
https://www.researchgate.net/project/Multicollinearity-Validity-Concentration-and-Prediction-Accuracy
My best to your research,
John
Yes, ridge regression is usually achieved by rescaling the covariance matrix of your independent variables prior to OLS regression. Strictly speaking you can do this rescaling of your independent variables regardless of what regression model you then apply.
I have some minor issues with John Morris here is a brief discussion of lasso and multicollinearity mixed in with other things in this attachment. Pay particular attention to references in that paper and those given here: https://www.google.com/search?q=Boos+adaptive+lasso+R&rlz=1C1CHBF_enUS874US874&oq=Boos+adaptive+lasso+R&aqs=chrome..69i57.15445j0j1&sourceid=chrome&ie=UTF-8
esp. those given by Prof Boos. The take-home message is that adaptive lasso does not allow correlated predictors in a prediction model and further gives an optimal solution for the predictors entered(Oracle property). Further all of these things can be done with an elastic net algorithm. See:
https://www.google.com/search?rlz=1C1CHBF_enUS874US874&ei=bZQeXq-bJ5ndtQbqpoTYAQ&q=elastic+net+regression+r&oq=Elsatic&gs_l=psy-ab.1.9.0i67l5j0i10j0i67j0i10j0i67l2.391400.397458..420510...0.2..3.141.2471.24j4......0....1..gws-wiz.....0..0i71j33i160j33i299j0i273j0i131i273j0i131j0j0i131i67.F6EK0juOy5A
I apologize for just seeing this further discussion now.
May the force be with you, D. Booth
- Badges
- Science topic
- Similar topics
- Mathematics
- Statistics
- Regression