Dear Ann,

Here is something about this problem. You can read more from:

What Are the Effects of Multicollinearity and When Can I Ignore Them?

Jim Frost,  2013

"How Problematic is Multicollinearity?

Moderate multicollinearity may not be problematic. However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret. Multicollinearity saps the statistical power of the analysis, can cause the coefficients to switch signs, and makes it more difficult to specify the correct model.

Do I Have to Fix Multicollinearity?

The symptoms sound serious, but the answer is both yes and no—depending on your goals. (Don’t worry, the example we'll go through next makes it more concrete.) In short, multicollinearity:

can make choosing the correct predictors to include more difficult.

interferes in determining the precise effect of each predictor, but...

doesn’t affect the overall fit of the model or produce bad predictions.

Depending on your goals, multicollinearity isn’t always a problem. However, because of the difficulty in choosing the correct model when severe multicollinearity is present, it’s always worth exploring."

http://blog.minitab.com/blog/adventures-in-statistics/what-are-the-effects-of-multicollinearity-and-when-can-i-ignore-them

Dear Ann,

Here is something about this problem. You can read more from:

What Are the Effects of Multicollinearity and When Can I Ignore Them?

Jim Frost,  2013

"How Problematic is Multicollinearity?

Moderate multicollinearity may not be problematic. However, severe multicollinearity is a problem because it can increase the variance of the coefficient estimates and make the estimates very sensitive to minor changes in the model. The result is that the coefficient estimates are unstable and difficult to interpret. Multicollinearity saps the statistical power of the analysis, can cause the coefficients to switch signs, and makes it more difficult to specify the correct model.

Do I Have to Fix Multicollinearity?

The symptoms sound serious, but the answer is both yes and no—depending on your goals. (Don’t worry, the example we'll go through next makes it more concrete.) In short, multicollinearity:

can make choosing the correct predictors to include more difficult.

interferes in determining the precise effect of each predictor, but...

doesn’t affect the overall fit of the model or produce bad predictions.

Depending on your goals, multicollinearity isn’t always a problem. However, because of the difficulty in choosing the correct model when severe multicollinearity is present, it’s always worth exploring."

http://blog.minitab.com/blog/adventures-in-statistics/what-are-the-effects-of-multicollinearity-and-when-can-i-ignore-them

Thanks a lot ! :) The comment about political powerlessness and participation correlating a lot makes sense. I checked the correlation and pearson is .298** (significant on the 0.01level)

That level of correlation makes sense.  But also keep in mind that with a correlation of around .3, you still have about 90% of the variability non-overlapping (i.e., .3 squared=.09, which leaves you .91 or about 90% of the variance unexplained).  Your main problem in analyzing the full model is getting enough power to overcome the multi-colinearity.  The piece from Beatrice (Jim Frost's piece) is really useful for this process.  Good luck!!

Power calculations are partly impacted by the size of the effect and partly by the amount of variance there is to be explained.  If you have serious multi-colinearity, it can reduce that amount of variance available to be explained.  The practical upshot is that the more correlation you have between predictors in your model, the larger a sample size you tend to need to observe significance.

    Keeping in mind all the caveats, of course, including the reliability of the measures, the stability of the estimates, and all the sources of error in your model.

Have you considered looking into Partial Least Squares (PLS) which overcomes some of the problems of multi-colinearity.  You will be able to easily find references for this methodology.  

We analyzed the impact of age and gender on attitudes towards mental disordered persons by means of different statistical methods. You can look it up.

Hello Ann. Another thought may have to do with your interaction. If you have an interaction term, have you centered it (subtracting the mean)? If you have not, this will quite likely give rise to multicollinearity between the interaction term and your independent variables used in the interaction term.

We never did this in class using SPSS, so I dont know how to center . 

In SPSS, do the following:

1) run descriptives on your interaction variable to find out what the mean is.

2) Under "Transform" you want "Compute"

3) This opens a new window.  Type your new variable name in the left-hand box, then in the right hand box, type in your old variable - the mean you just found.

Or, you can open a syntax window and just type this:

COMPUTE newvar=oldvar - XX.

EXECUTE.

where "newvar" is the new variable name, "oldvar" is the old variable name, and XX is the mean you found in step 1.

Hello Ann and Julia,

I believe you will need to mean centre the two IV terms first and then calculate the interaction term. You will still need to include the original IV terms in your regression analysis (uncentred), but he interaction term is the one computed from the centred IV's. 

Your thoughts, Julia?

First this is the norm for interactions. You are fitting a model with X1, X2 and X3 (where X3 is X1 x X2) as predictors. X1 x X2 tends to be correlated with X1 and X2. For instance if X1 = 3, X2 = 4, X3 will be 12 (a positive number that increases in magnitude as X1 or X2 increase).

In this model the test of the interaction is the change in model fit with and without X3 in the model. This test is unaffected by the collinearity X3 and X2 or X2 and X1 (because changes in coefficients and standard errors are offset by changes in collinearity). Thus there is no need to center any variables if you want to test the interaction. Likewise the predictions of the model are unaffected by the collinearity of the predictors.

What centering does is to rescale X1 and X2 so that their individual coefficients may be easier to interpret. (It makes the coefficients for X1 and X2 similar in both the interaction and no interaction model) This is sometimes desirable (e.g., for predictors on an arbitrary scale) and sometimes less so (e.g., for categorical predictors). In essence the decision to center depends on whether it makes the model easier to interpret (and will vary from person to person and data set to data set).

One confusing point is that X! and X2 may have different coefficients and significance on the centered and uncentered interaction model. This is a feature not a bug. In both models the impact of X1 depends on X2 and vice versa, thus it shouldn't be surprising that changing when X1 is zero changes X2 (or vice versa). It tells you that is dangerous to interpret X1 and X2 in isolation when you have an interaction ...

Dear Ann,

It would help the discussion a lot if you wrote out your multiple regression model in full detail.

For a "continuous" predictor, the first step should be to center it (not necessarily at the mean; sometimes a suitable round value is better).  Centering those predictors will make the intercept interpretable. It may also reduce collinearity (e.g., in some data, a predictor may resemble a constant).

Centering does not rescale a predictor.  As a result, centering may have no effect on the coefficient of the predictor.  When the model contains an interaction of two continuous predictors, centering those predictors and using the centered predictors to define the interaction term will change the coefficients of the two predictors (but not the coefficient of the interaction term), simply because the set of predictors is different.  Building on Thom Baguley's example, centering X1 at a1 and X2 at a2 and defining X4 (renamed to make clear the difference from X3) by X4 = (X1 - a1)*(X2 - a2) yields the three predictors (X1 - a1), (X2 - a2), and X4.  To see how the coefficients of X1 and X2 differ from those in the model whose predictors are X1, X2, and X3 = X1*X2, multiply out X4 and compare those coefficients.  The general idea is that the definition of the coefficient in a multiple regression includes the set of other predictors in the model.

A large VIF indicates the presence of collinearity involving that predictor, but it doesn't give more information.  One useful approach to diagnosing the details of a collinearity relation (i.e., the specific variables involved) uses the "regression coefficient variance decomposition," described in the book by Belsley, Kuh, and Welsch, Regression Diagnostics (Wiley, 1980) and available in some statistical software. 

Dear Ann,

I published an article at the end of 2013 in which I developed an approach that deals with the issue you raise. This new approach, which I call Sequential Residual Centering (SRC), eliminates the portion of multicollinearity called "inessential ill-conditioning," which is the multicollinearity among the component terms of an interaction. For instance, for the highest-order interaction xwz, inessential ill-conditioning would be the multicollinearity among the following related terms (x, w, z, xw, xz, wz, xwz). [Of course, these terms may also be characterized by "essential ill-conditioning," which refers to their multicollinearity with other specified predictors that are not also component terms of the interaction. The portion of multicollinearity that contributes to "essential ill-conditioning" cannot be eliminated in SRC, however I also reported in my article an extended version of SRC I created (SRC-CS) that does also allow conditioning essential ill conditioning from control variables or secondary predictors].

When VIF values are inflated, my SRC approach obtains the same regression slope values but lowers both the standard errors and the VIF (and therefore the p values) for the component terms of an interaction, which improves the sensitivity and statistical conclusion validity of moderated regression (i.e., reduces Type II error). I conducted simulations which validated the SRC under a variety of conditions. SRC and SRC-CS are relatively easy to understand and conduct using any regression software.

The citation for my article is: Francoeur, R. B. (2013). Could sequential residual centering resolve low sensitivity in moderated regression? Simulations and cancer symptom clusters. Open Journal of Statistics, 3, 24-44. You can locate and download it under my profile on ResearchGate (https://www.researchgate.net/profile/Richard_Francoeur) or in the December 2013 issue of the Open Journal of Statistics (http://dx.doi.org/10.4236/ojs.2013.36A004).

P. S.  I also published two methodology articles that provide further explanation and apply SRC to detect cancer symptom interactions in an open access oncology journal. The citations for these two articles are: 1) Francoeur, R. B. (2015). Using an innovative multiple regression procedure in a cancer population (Part I): Detecting and probing relationships of common interacting symptoms (pain, fatigue/weakness, sleep problems) as a strategy to discover influential symptom pairs and clusters. OncoTargets and Therapy, 8, 45-56; and 2) Francoeur, R. B. (2015), Using an innovative multiple regression procedure in a cancer population (Part II): Fever, depressive affect, and mobility problems clarify an influential symptom pair (pain–fatigue/weakness) and cluster (pain–fatigue/weakness–sleep problems). OncoTargets and Therapy, 8, 57—72. You can locate and download these two articles under my profile on ResearchGate (https://www.researchgate.net/profile/Richard_Francoeur) or in the January 2015 issue of OncoTargets and Therapy (http://dx.doi.org/10.2147/OTT.S66465 and http://dx.doi.org/10.2147/OTT.S68859).

according to PAUL D. ALLISON, when "the high VIFs are caused by the inclusion of powers or products of other variables," "you can Safely Ignore Multicollinearity." please see http://statisticalhorizons.com/multicollinearity. However, can anybody provide a formal reference?

Liqun Luo: A comment by Pat Rubio Goldsmith in the Paul D. Allison page you cite states: "I seem to recall from an old Hanushek book that multicollinearity does not bias coefficients; it inflates their standard errors. These large standard errors make p-values too large. That is a problem when the p-values go above a threshold like .05, but otherwise, the inflated standard errors don’t change the interpretation of the results. So, in addition to the comments you made above, multicollinearity does not usually alter the interpretation of the coefficients of interest unless they lose statistical significance."

These comments are borne out in my study that proposes and tests Sequential Residual Centering (SRC), referenced in my posts above.  The study provides an example based on real data involving cancer symptoms in which the interaction was not statistically significant (p > .05).  However, when SRC was applied, the interaction became statistically significant (P < .05).

Paul D. Allison supports his argument by showing that the use of mean centering does not affect the p values when x2 or xz are correlated with their lower-order derivative components (x or z).  It has been demonstrated that MEAN centering does not reduce multicollinearity.  However, in my study, I show that the use of RESIDUAL centering, specifically SRC, does reduce the p values when the variance inflation factor (VIF) is inflated.

Dear Ricahrd, Thank you very much for your response. Probably most researchers are only interested in the application of statistical knowledge. However, probably I will cite your works.

People usually are lazy. And most researchers are not statistical experts. We also have a lot of more important things to do. So, when we encounter this kind of things, we might cite Robert Friedrich (1982, In Defense of Multiplicative Terms in Multiple Regression Equations, AMERICAN JOURNAL OF POLITICAL SCIENCE, 26(4)) to show  that multicollinearity involving interaction terms is only a negligible problem?

Liqun, I respectfully disagree that multicollinearity involving interactions is a negligible problem.  It can mean the difference between considering an interaction to be statistically significant (p < .05) or not, or whether it is highly significant versus significant.  The SRC approach is not difficult.  I think you would show reviewers that you gave the matter serious thought, which may lead them to believe in the soundness of your work.

Thanks to everyone's comments on this important problem. I encountered it myself and still speculate whether I should use Sequential Residual Centering while Time term has got 14 VIF and intercations are beyond 50?

Why not run it separately both ways (with and without using Sequential Residual Centering) to determine whether the level of statistical significance changes? It would allow you to see whether retaining the multicollinearity would lead you to draw different conclusions about whether there is/are statistically significant effects or the probability level at which any statistically significant effects occur (e.g., p < 0.10 versus 0.05 versus 0.01).

Thank you for your clear answer. I haven't changed my categorical variables into dummy variables, and I think I would have to do that before reanalysing again.