I have been trying to understand exactly how using LASSO to build a model deals with the issues of stepwise model building. I would love to hear if I'm on the right track with this interpretation:
LASSO limits the sum absolute value of coefficients in a regression model, which prevents any predictors from being particularly influential. This deals with issues of overfitting in stepwise model building, in which predictors that are especially influential in the sample data will get high coefficients. This is a problem because these especially influential predictors are likely to only be so influential in that sample of data; in the real world it is less likely that any predictors really deserve coefficients that high. Because of this, an overfit model with extremely high coefficients on some variables will do a poor job predicting a new sample of data (which is unlikely to have those predictors be as influential as they were in the original sample).
(I know another benefit of LASSO is that it uses cross validation to find values of lambda, but I'm interested in the specific topic I mentioned above.)
Many regression methods limit the magnitude or norm of the coefficients to constrain the solution space. I understand the main benefit of LASSO to be penalising very small coefficients (setting them to zero if small enough) based on the assumption that these are more likely to be low signal to noise and therefore add more noise than signal to the prediction when applying to independent data.
To my knowledge, LASSO not only limits the sum absolute value of coefficients in a regression model by shrinking the extremely high coefficients – which are likely to be overfitted and won't be replicated in an independent sample – but also by setting very small coefficients to zero, as James mentioned, and by setting coefficients of variables to zero, which are highly correlated with other variables in the predictor set (to deal with the problem of multicollinearity).
Summed up, after shrinkage, the high coefficients should remain the highest in the set of predictors and therefore 'especially influential' (except for the highly correlated ones, because all but one of them is set to zero). Results of a recent analysis I performed, comparing the estimates of a bootstrap ranking LASSO and a logistic regression using glm, point in this direction.
Hi David,
Please share your work upon Lasso. It may be useful to me. Thanks
- Badges
- Science topic
- Similar topics
- Philosophy
- Esthetics
- Interpretation