Hi all,
I am dealing with data with sevaral features and many of them are highly correlated with each other as well as with dependent variable.
In my research on this topics, I found that multicolinearity is harmful for regression problem and may not end up with good model. I got some suggestion that if the features are highly correlated then we have to remove them using VIF criterion.
But, logically when I think of removing correlated features from my analysis how can expect better model as I am not considering all the available information.
Is there any logical explaination or mathematical explaination is available for the above question?
Also, I am thinking that each features are somehow related to any of the other (May be nonlinearly) in that case, do we have problem of multicolinearity ?
There is a good explanation here:
https://www.researchgate.net/post/How_to_deal_with_Multicollinearity
and here:
Article A Comparative Study On Some Methods For Handling Multicollin...
In Multicollinearity you need to set the threshold at which you want to remove the column. If the threshold is low then valuable data might be lost and if threshold is high then Multicollinearity might not be addressed. It's a trade-off between loss of data and curse of dimensionality... A rigorous mathematical proof for this is beyond me..
The "mathematicaL" explanation is to examine the formula for the partial regression coefficient. It contains the covariance between the two independent variables, and if this is quite high, it will dominate the estimation of the coefficient. At the most extreme case, the two independent variables are identical, so neither one can make a separate contribution.
In my opinion, the best approach is to choose the most theoretically relevant variable from each set of highly correlated variables and use that one. This may well not cost you as much as you might think, because once you enter one of the variables, adding any of the other variables will amount to putting "the same thing" into the equation, so you R-sq is unlikely to increase.
Because it undermines the statistical significance of an independent variable. Other things being equal, the larger the standard error of a regression coefficient, the less likely it is that this coefficient will be statistically significant.
I recommend non-linear analysis of network models according to the network theory that allows to include partial correlation values.
- Badges
- Science topic