Hi Saadia you need to deal with these two problems separately and sequentially:

1) First, If your dataset has a strong multicollinearity, then remove those variables with a correlation value above 0.6 (or so). A more elegant procedure will be to assess the variance inflation factor (VIF) for each variable, and select only those variables with a VIF below 10 (other recommends VIF < 4, the cut-off is really arbitrary). Check this highly cited paper on the topic:

https://www.researchgate.net/profile/Robert_Obrien8/publication/226005307_A_Caution_Regarding_Rules_of_Thumb_for_Variance_Inflation_Factors/links/54d0f2620cf298d656695641.pdf

Another option would be to run a principal component analysis (PCA) on you predictor variables, with should remove multicollinearity, and then run the regression on the first axes than accounts up to 95% of explained variance or so. Of course this makes the data interpretation a bit more cumbersome, because each PCA axis can be related to several of your original variables. Hence, checking the correlation between PCAs and the original variables is highly recommended. 

2) Once you have accounted for multicollinearity you may procede with regression with robust standard errors or any other method (e.g. GLM with gamma errors).

Best regards

Marcelo

Article A Caution Regarding Rules of Thumb for Variance Inflation Factors

Saadia -

This is an interesting question.  Marcelo may have a good approach.  I suppose there might be several ways to deal with this, perhaps either sequentially or simultaneously.  Not knowing your data or purpose or subject matter, which all may impact what might be good for you, let's look at the problems of heteroscedasticity and multicollinearity in general. 

For collinearity (different word but same as multicollinearity), the degree is going to be somewhat nebulous, as the degree of impact is not generally very clear.  Principal components may not leave you with results that are very clearly interpretable.  There are up and down sides to everything.  If removing a specific regressor, or combination of regressors, appears to greatly reduce your collinearity, without substantially  biasing results, you might want to do that, but variables can be so interrelated that it may be unclear as to which combinations of regressors are 'best.'  

I suggest you try a few reasonable models, from the standpoint of subject matter theory, hold out some of your data as if you did not have it, and test your models to see which ones would have "predicted" those data best. 

If you think a ridge regression best, by all means, give it a try, though I do not know how best to pick lambda. 

Anyway, if you look at the following from a Pennsylvania State University site, under "Ridge Regression," it shows an estimator for beta, using ridge regression for OLS regression:

https://onlinecourses.science.psu.edu/stat857/node/155

Compare that to the estimator for beta for weighted least squares, not very far from the beginning in the following:

 https://onlinecourses.science.psu.edu/stat501/node/352

If you have heteroscedasticity, that is quite natural (why would a huge response have the same standard error as a small one?) and can be handled as the natural part of the error structure that it may well be.  The one regressor case through the origin is shown, hopefully for clarity,  derived at the bottom of page 2 in https://www.researchgate.net/publication/263036348_Properties_of_Weighted_Least_Squares_Regression_for_Cutoff_Sampling_in_Establishment_Surveys

An often robust degree of heteroscedasticity can lead to a regression weight of w=1/x.  You can estimate the coefficient of heteroscedasticity a number of ways.  One method is called the IRLS (iterated reweighted least squares).  Your software might have that.   The coefficient of heteroscedasticity can be examined by looking at the results of residual analysis graphics, and carrying them further:

 https://www.researchgate.net/publication/263809034_Alternative_to_the_Iterated_Reweighted_Least_Squares_Method_-_Apparent_Heteroscedasticity_and_Linear_Regression_Model_Sampling

 https://www.researchgate.net/publication/263032446_Weighting_in_Regression_for_Use_in_Survey_Methodology

So, if you are using ridge regression, and you have heteroscedasticity, then it would seem you need a reasonable lambda, and regression weight, w.   If you are relying on software such as R or SAS or any other package to help, then you need to see if a given routine will let you input both lambda and w.  Also please note that you have to be sure that your software is using those same symbols.  If you find a procedure/routine that you think will do this, you'd best check the software documentation, to see that it really is doing what was depicted by the combination of formulae in the two Penn State links above, with both lambda and W in the 'formula' for the beta estimator that you want. 

If you do not have software for this, for multiple regression, then that would be problematic, as deriving and programming your own version would be very involved and prone to error.  But using statistical software can be error prone as well, in that you must be sure that your software is doing what you want it to do. 

This is just my guesswork, but if you do find software that will let you input both lambda and w, or find some way to 'make' your software do that, it might be helpful and interesting to others for you to post that here. 

I hope that at least some of that was helpful and not too far afield from your question. 

Cheers - Jim

Article Weighting in Regression for Use in Survey Methodology

Article Properties of Weighted Least Squares Regression for Cutoff S...

Conference Paper Alternative to the Iterated Reweighted Least Squares Method ...

Hi Saadia,

I am only dealing with colinearity in this post, as I think I don't have enough experience in dealing with heteroscedasticity. I read in the answers above that both PCA and ridge regression can be used to tackle this issue and I agree with this. However, both methods introduce a loss of interpretability in variable importance, as they focus on using principal components. 

I consequently suggest another method : LASSO regression by Robert Tibshirani : see http://statweb.stanford.edu/~tibs/lasso/lasso.pdf . As for the choice of the penalization parameter, I suggest you to use cross-validation methods (Leave One Out CV if you have a small sample (#10^2) or K-folds CV if your sample is quite large).

Note that the LASSO regression can be combined with the Ridge Regression and becomes then what is called "Elastic Net Regression", but this point is not essential here.

What is important is that Lasso acts like a net (which explains why the Elastic Net term : Net from LASSO and Elastic from Ridge), as it uses L1-norm for beta penalization. Consequently, as the penalization cost goes up, some beta-coefficients will be adjust to 0, and then some others, and so on. This method is great for variable selection and is consequently heavily used in (for example) statistics applied to genomics because it allows one to identify the few genes over millions that are responsible for a phenomenon. Of course, L1-penalization also allows one to tackle the issue of (multi)collinearity by which you seem to be interested here.

I hope this helps a little,

Best regards !

thanks a lot all of you. Your feedback really helps

What if you were to use simultaneous equations method to tackle multicollinearity; and then use robust standard errors within that to tackle heteroscedasticity?