yes you can do it or you can find VIF witch variable have high VIF you first remove that variable and if you want to see x3  & x4 simultaneously effect on y then you first make there PCA that uncorrelated then run OLS

Thanks. I had actually tried PLS regression. However, I just want to know whether building separate models is a right method. I think even PLS estimates are a little biased.

Srijith -

From the point-of-view of having worked with a great deal of continuous data, I find your two equations very interesting.

Consider the extreme case of multicollinearity where no additional information is found when looking at both x3 and x4. An example I heard once was something like, Suppose one was temperature Celsius and the other temperature Kelvin. In that case, your two equations would be equivalent. In the extreme case that X3 and X4 are uncorrelated, you would have had no collinearity, but I don't see that happening. I think collinearity is more a matter of degree than "yes" or "no."

One thing you might try that might relate to your research is to combine the two estimates by stein-like/shrinkage factor-like coefficients, say (1-a ) times one and a times the other, where 0

Sir, thank you for the response. It gave me more insights to the issue of collinearity. But in my case it seems I have collinearity between dummy variable and a continuous variabe (and collinearity between categorical variable and continuous variable). I have already built my final models. I would like to know if building separate models may give reasonably correct estimates. They are not much correlated, but when I add both in the model, the t stats are falling down. It is a tricky situation where I want to know the individual effect of both, (which is obviously different) rather than improving my model predictability.

Hi Srijith,

I published an article at the end of 2013 in which I developed an approach called Sequential Residual Centering (SRC) that deals with the issue of conditioning away multicollinearity in interaction terms, however a variation of this approach (SRC-CS) can be used in regressions without interaction terms.  SRC and SRC-CS are relatively easy to understand and conduct using any regression software.  I describe the approach in my answer to another question regarding multicollinearity involving interaction terms (https://www.researchgate.net/post/Is_it_a_problem_to_have_multicollinearity_with_interactions). 

Here let me point out the SRC-CS allows conditioning away of "essential ill-conditioning," which is the multicollinearity between two or more predictors that are not involved in an interaction (which is your situation).  SRC-CS allows conditioning essential ill conditioning from control variables or secondary predictors so that you can reduce the inflated VIF value by obtaining the UNIQUE predictive estimate of X3 and of X4 (after partializing out the effect of the other predictor).  First, you partial out the effect of the other predictor by estimating the following residualizing regressions (without the intercept term):

1. X3 = c1*X4 + d[X3]

2. X4 = c2*X3 + d[X4]

d[X3] and d[X4], which are the respective error terms of the two residualizing regressions, are the distribution of the respective predictor after partializing out the effect of the other predictor.  (Note that since there is only one predictor in each equation, inflated VIF values are not an issue).

Thus, using d[X3] and d[X4] in place of X3 and X4, the regression coefficients based on the UNIQUE predictive information within X3 and within X4 (i.e., b3 and b4) can then be estimated using:

Y = b0 + b1*X1 + b2*X2 + b3*d[X3] + b4*d[X4] + error

The VIF values fall dramatically in this equation compared to your original raw regression.

You can then rearrange the estimated residualizing regressions to obtain:

1a. d[X3] = X3 - c1*X4

2a. d[X4] = X4 - c2*X3

and plug in these estimated values into the estimated regression:

Y = b0 + b1*X1 + b2*X2 + b3*(X3 - c1*X4) + b4*(X4 - c2*X3)

This is equal to:

Y = b0 + b1*X1 + b2*X2 + (b3 - b4*c2)*X3 + (b4 - b3*c1)*X4

Thus, the regression coefficient for X3 equals b3 - b4*c2, and the regression coefficient for X4 equals b4 - b3*c1.  In contrast to the regression coefficients that are based on the UNIQUE predictive information within X3 and within X4 (i.e., for d[X3] and d[X4]), these regression coefficients for X3 and X4 are based on all of the predictive information in each variable, including the information shared by both variables that constitutes multicollinearity.  (You may want to base tests regarding statistical significance on b3 and b4 from the earlier equation with the UNIQUE predictive information within X3 and X4.)

This is the approach of SRC-CS.  Using SRC-CS provides the unique estimates of X3 and X4 in contrast to the raw regressions that you specified in which the shared multicollinearity between X3 and X4 are attributed entirely as part of the unique effect of each variable (since the other variable is not specified in the same equation).  The choice between these two regression specification options would depend upon the purpose of your research.

The citation for my article that describes SRC and SRC-CS is:  Francoeur, R. B. (2013). Could sequential residual centering resolve low sensitivity in moderated regression? Simulations and cancer symptom clusters. Open Journal of Statistics, 3, 24-44.  You can locate and download it under my profile on ResearchGate (https://www.researchgate.net/profile/Richard_Francoeur) or in the December 2013 issue of the Open Journal of Statistics (http://dx.doi.org/10.4236/ojs.2013.36A004).

Good luck in your work,

You can also use a Ridge Regression model to deal with collinearity among your IVs by imposing a penalty on their size. With this method there is no need to remove variables but the disadvantage is that this method is computationaly "heavier" in my opinion. I would recommend using principal component analysis.

Richard's approach above sounds quite reasonable to me, and worth considering.  I had not heard of it, and did a little internet research.  There, however, I found the following criticism:

In the first paragraph of page 42 in

 "What residualizing predictors in regression analyses does (and what it does not do), Lee H. Wurm, and Sebastiano A. Fisicaro, Journal of Memory and Language 72 (2014) 37–48, Elsevier,

  https://www.google.com/url?sa=t&source=web&rct=j&url=http://clas.wayne.edu/Multimedia/wurm/files/Wurm%2520%2526%2520Fisicaro%2520(2014).pdf&ved=0CC8QFjAFahUKEwjM_MjUkITJAhVDNT4KHcZvAP8&usg=AFQjCNGebfJuTDFSiPzEhA0TjGO78YobLw&sig2=288MockwyEaFO3DGr8nDCQ

they say,

"However, the correlation between two variables residualized against each other will always have the same magnitude as that between the original variables, with the opposite sign. ... Such an analysis

thus does not reduce collinearity, and worse, contains neither of the original predictors of interest."

That sounds like a situation to be avoided, though I'm not sure if it relates directly to the question by Srijith, but it seems to me that it does.  

They also say that "... it [very much] matters which variable is residualized." (p. 44).

On p. 45, however, it says "First and foremost, it produces an intercorrelation between predictors of 0, which was of course its desired effect."  And it indicated other good news, but it then goes on to say that it presents "conceptual difficulty."

Hopefully I have not too badly misrepresented anything here, as this is new to me.  I just took a quick look at the Wurm and Fisicaro paper.

I do not know whether or not the criticism in that paper is valid, but it might be checked out empirically.  That article has some such evidence, though I have not checked it out myself, and think it often hard to say exactly what someone else did, so if anyone has more first hand knowledge and can explain this, it would seem to be of interest.  They appear to have looked at some more complex models, but I'd like to hear more about simpler cases.  I'd like to more clearly understand the basic concepts relevant here.

(Note that when considering evidence, correlations and standard errors may be of interest, but p-values are not so useful, and generally misinterpreted, and R-square is volatile and changes definition from model to model.)

Does anyone want to explain this?  Maybe a short, clear synopsis of what residualizing predictors can and can't do here for the question asked by Srijith?