Dear Mohsen Ahmadkhani,

Here is an overview of various problems that might occur in multiple regression and their possible measures.

Problems That May Occur in Time Series Multiple Regression:

1. Multicollinearity:

If one independent variable is excessively linearly correlated with another independent variable, then it will be impossible to determine their separate influences. The problem is with the data and not with the regression model itself and will be signified three schemes

i. high R2 with low values for the t statistics,

ii.  high values for simple correlation coefficients between the independent variables, 

iii. regression coefficients that are sensitive to model specification when both variables are included.  A variable may even take the wrong sign.

Multicollinearity may not be a problem in forecasting but it will be a problem is simulation.

Solution to the problem is to either (a) combine variables or (b) eliminate one of the variables or, possibly, (c) increase the sample size.

Stepwise regression will also eliminate the problem and select the “best” predictor variable to enter when other independent variables are present.

2.  Heteroscedasticy. 

The error terms are assumed to have a constant variance, but if the variance depends upon the value of X, then heteroscedasticy exists.          

The Goldfeld-Quandt test for heteroscedasticy may be used.

The correction for heteroscedasticy requires the use of (a) weighted least squares, (b) transformed variables, or (c) respecification of the model.

3.  Autocorrelation. 

Error terms are assumed to be independent over time.  Autocorrelated error terms are the most important consideration for a practicing forecaster.

The Durbin-Watson statistic can be used to test for autocorrelation problem.

Solutions include (a) finding an important omitted variable (b) transforming the variables based upon generalized least squares and (c) introducing a lagged value of the dependent variable on the right hand side of the equation.

I hope this will be helpful to you.

All the best !

Dear Mohsen,

I think that when predictors are correlated, you can substitute the dependent  variable in terms of the independent variables after finding the co relation equation and get a new  equation with reduced independent variables.

Correlated independent variables is not a problem if you only care about predicting an outcome; it's a problem when you want to estimate the unique contribution of each independent variable. If you only care about predicting the outcome, you might be able to ignore this problem.

If it is a problem, there are lots of ways to handle multicollinearity, including using only a subset of the independent variables, orthogonalizing some of the variables, or turning them into a composite of fewer variables (using e.g. PCA). There are lots of resources online discussing these options.

Hi, the problem with correlated variables is that it generally introduces an increase in the MSE of prediction on your test set due to a high process error. I suggest you have a look at ridge/lasso/elastic net regressions that penalize the "size" of the beta coefficient by adding a cost in the usual norm 2 you try to minimize when solving an OLS problem.

These methods will add a bias in the estimation of your coefficients that will lead to an increase in the estimation error, on the other hand your process error will decrease, especially if your independent variables are highly correlated.

Hope this helps,

Best regards.

Dear Mohsen,

You can use multiple linear regression, if four principal assumptions are met. There are four principal assumptions which justify the use of linear regression models. See, for example:

http://people.duke.edu/~rnau/testing.htm

Pay attention to the first assumption (linearity and additivity of the relationship between dependent and independent variables), and check the distribution of the dependent variable for normality. If the assumptions are not met, it is necessary to proceed to a non-linear regression.

Yes I agree with Mr. Stephen. There are various ways to handle multicollinearity, in fact, you should also check the nature of variables or data, then choose the best fitted regression model. 

Dear Mohsen Ahmadkhani,

Here is an overview of various problems that might occur in multiple regression and their possible measures.

Problems That May Occur in Time Series Multiple Regression:

1. Multicollinearity:

If one independent variable is excessively linearly correlated with another independent variable, then it will be impossible to determine their separate influences. The problem is with the data and not with the regression model itself and will be signified three schemes

i. high R2 with low values for the t statistics,

ii.  high values for simple correlation coefficients between the independent variables, 

iii. regression coefficients that are sensitive to model specification when both variables are included.  A variable may even take the wrong sign.

Multicollinearity may not be a problem in forecasting but it will be a problem is simulation.

Solution to the problem is to either (a) combine variables or (b) eliminate one of the variables or, possibly, (c) increase the sample size.

Stepwise regression will also eliminate the problem and select the “best” predictor variable to enter when other independent variables are present.

2.  Heteroscedasticy. 

The error terms are assumed to have a constant variance, but if the variance depends upon the value of X, then heteroscedasticy exists.          

The Goldfeld-Quandt test for heteroscedasticy may be used.

The correction for heteroscedasticy requires the use of (a) weighted least squares, (b) transformed variables, or (c) respecification of the model.

3.  Autocorrelation. 

Error terms are assumed to be independent over time.  Autocorrelated error terms are the most important consideration for a practicing forecaster.

The Durbin-Watson statistic can be used to test for autocorrelation problem.

Solutions include (a) finding an important omitted variable (b) transforming the variables based upon generalized least squares and (c) introducing a lagged value of the dependent variable on the right hand side of the equation.

I hope this will be helpful to you.

All the best !

Hello Mr. Swain,

Very well defined. Could you please define the meaning of Error term?

Thanks in advance.

Dear Shweta Mehrotra,

Thanks for your appreciation.

Regarding your question, Error terms (you can also say residuals in this case) is the difference between the regression line value and the actual value. Whenever you are drawing/fitting a regression line, it will never touch all the points. So, there is a difference between the value on regression line and the corresponding actual point. That is the error term (Residual, to be more accurate) for my answer above. Lesser the residuals, more accurate is the regression model.

I hope your doubts will be cleared by this.

All the best !

sir namasthe

let us take an example for existance of touching all points in a line. Assume t hat two variables are ht. and wt. and there is a correlation as + or - 1 between them...

Dear Rayalu,

Obviously that is possible, but that is an an ideal case.

In general, hardly we can see cases where such thing happens. When we plot regression (that too multiple regression), it will hardly fit all points exactly. Correlation of +1 or -1 is of course possible, those are the limits of correlation. But it almost never occurs (may be once in a million or even lesser).

I hope this clarifies.

All the best !

Thanks Mr. Swain for your answer. Now, I got it.