In Multiple Regression, what do you do in a variable selection process when some model(s) with excellent fit (High adjusted R-squared, low Mean Square Error, Low Mallow Cp value) display a high level of multicollinearity (several high Variance Inflation Factors), while models with manageable levels of multicollinearity have much lower values for adjusted R-squared?
Would you run Ridge Regression (more complicated to many users), or would you inspect the regressor variables (or independent variables) with the goal to make the models shorter by eliminating some of the X's, or what else would you do in such a situation?
How damaging to the regression is a high level of multicollinearity, and how do you balance between fitting and prediction; between over-fitting and under-fitting?
This situation happens often to me. Thanks for your feedback! We can all learn from each other here.
Hello, Raid! I think your problem is a very interesting one. I suggest you read about the Successive Projections Algorithm. Perhaps it may be very helpful for you. If you wish, you may take a look at my profile and see some papers of mine that describe this algorithm.
Best wishes and good luck with your research!
Dear Prof. Raid Amin! You are right. The SPA is an iterative algorithm that consists of 3 phases. In phase 1, the variable chains are created. In phase 2, the variable chains are evaluated according to a error measure. Finally, the phase 3 discards those variables that do not contribute significantly to the predictive capability of the model.
On the other hand, I have demonstrated that the Firefly Algorithm may be a very good solution for variable selection. If you wish, I suggest you to read a paper of mine called "Parallelization of a Modified Firefly Algorithm...".
If you have any question, please do not hesitate to contact me directly. I would be grateful if my answers were appreciated (VOTE UP) by you to help increasing my RGScore.
Best regards!
If you have a high VIF, you can get coefficients that change directions and magnitudes. Oddly, the model may still be very valid. If you try to estimate the response of something, you might(I heard should) get good results. If you try making a statement like, "By increasing varible X, you should...." it will probably be really bad.
I have been fortunate to have as a graduate student Professor Raymond Myers in several courses on regression. People like Douglas Montgomery and Geoffrey Vining and William Woodall (and so many others) had him as their professor too. An excellent text on regression is by Montgomery, Peck, and Vining. In the 5-7 courses that I had with him, he taught me many things about multicollinearity in regression.
When you study the role of eigenvalues and the hat matrix in multicollinearity, and you see the many diagnostics that measure the effects of multicollinearity, you can see that while fitting is not much affected (in fact, it can "improve"), prediction suffers greatly.
I don't know any of the authors of these four texts, Predrag. Is anyone of them a Statistics Professor at a Statistics Department? It could be that applied sciences have a different perspective on the seriousness of multicolinneatrity than the statistics discipline itself. Multicolinearity in statistics is similar to singularity of matrices in Linear Algebra. A singular matrix has no inverse. This is a problem. The views of some authors of texts do not necessarily create "facts" or "truth". Sometimes, they express personal opinions.
In Statistics, we may encounter near-singularity, where some X's are strongly associated with each other. This results in a X'X inverse matrix with some diagonal elements (the reciprocal of the eigenvalues) that are very large, and then the properties of estimators and variances become very poor. These are solid facts that can be shown analytically. These facts are usually taught at Ph.D. Statistics Departments.
.
Dear Raid, since statistical software has made us lazy, i should propose you to do the "dirty work" by hand. So:
1)take the vectors of all X's and store them
2)then build the submatrices with columns subsets of them
3)compute with a reliable software the rank of those submatrices
4)write down the observed non full-rank submatrices
5)decide which variables are really independent
I know that it seems difficult, but multicollinearity essentially means that the corresponding matrix is not a full - rank one.
That's all.
Demetris,
When you have a large number of X's, the approach that you have suggested will be very time consuming to do. Say, k=10 X's, and say that there is severe multicollinearity between several X's, you would need to create many submatrices to be able to correctly identify the multicollinearity. I think that it is better in such a case to use an advanced software package to obtain colinearity diagnostics.
Raid, of course you can use a software, but for 10 variables I think it will bother you more than doing it by hand. If the variables were 50, then the situation changes.
That's very nice, so actually you have answered your question. Well done, OK.
I think when something we do it 'by hand', then we can capture all of its dimensions. Programs must be tools and not 'gods', imho.
To detect multicollinearity you can observe the correlation and partial correlation matrix of the independent variables in the model. High correlation and partial correlation coefficient is an indication of multicollinearity among such independent variables. Another option is perform auxiliary regression with the independent variables. With this method the significance of the overall model is an indication that the ith independent variables is a linear combination of the other independent variables justifying multicollinearity. To solve such problem one can perform subset regression and then select the best model among the fitted models.
- Badges
- Science topic
- Similar topics
- Mathematical Sciences
- Control Theory
- Control Systems