Ridge regression may solve your problem. Also, you may use component analysis (PCA, ICA). Or may be feature selection, it depends on your data. What is the dimension of your data matrix?

Dear Gokmen Zararsiz!

Thanks for adding the answer.

I have 28 sugarcane genotypes and 16 characters. PCA reduced the characters to 9 (3 principal components explaining 75% of the total variation). Have further to estimate path coefficients. Could you explain ridge regression a bit further please.

It is a method works for the datasets which have multicollinearity. If multicollinearity exists in a data, the least square estimates are unbiased but their variance is larger than its expexted real value. This method adds a small bias to overcome this problem for more trustable results. NCSS software has this option to apply. I think SAS also has.

But your number of variables are larger than ypur observations. Thus, I believe that dimension reduction techniques such as component analysis will be a better idea!

Hope it works.

I have got 16 variables and 28 genotypes. Do we have to use the transformed data given by ridge regression technique or something, as you say it incorporates some bias in the data. I have already reduced the variables from 16 to 9 using PCA. So that leaves me with 9 variables. What I mean to say is how to go further from ridge regression? Thanks.

Oh OK, so the characters are your variables. But still seems that the number of observations are so small for this kind of predictive analysis.

If you solved the multicollinearity with PCA, you dont need to use ridge regression. You can go on with linear regression analysis.

Due to your small sample size, it will be better to use less principal components (maybe 4-5, considering the total variance explained) and apply linear regression if the multicollineartiy problem is solved.

Actually, What I want to do is Path Coefficient analysis which gives us direct and indirect effects of different characters on yield. It uses standardized partial regression coefficients. Now the problem is that due to this multicollinearity problem the residual goes on to take negative values and gives very high direct and indirect effects thereby making the problem worse.

I used SAS procedure for detecting multicollinearity and found high values for a pair of characters and removed both of them from correlation table. But this didn't solve the problem. I used threshold of 10 for VIF in SAS.

I would take two steps.

1) Run PCA with orthogonal rotation to derive meaningful Factors from your collection of variables---From your comment you can reduce to just 3 factors which should be groups of reasonably interpret-able variables.

2) Use the factor scores derived in step 1 as variables in your path and or regression analysis. The results will provide the path coefficients for each of the three Factors.

The additional step of rotation to factor scores usually provides much more interpretable factor loadings than PCA which simply generates independent scores that usually are uninterpretable.

The link below provides a nice overview contrasting PCA and FA and derivation of new variables.

http://www2.sas.com/proceedings/sugi30/203-30.pdf

Respected Dr John Kern!

Thanks for the insight into the problem. I applied PCA prior to path analysis. It gave me 3 principal components (with 9 characters loading on them). Then created new variables using the eigen vectors of the principal components and the raw data.

What I understand from you answer is that now I proceed with path analysis with these newly created variables? How will i have to interpret them as I will have to go on with development of selection indices as well after that.

What I did was that I used the characters (variables) extracted by PCA into Path analysis.

Using the extracted variables is one option, but often the multicolinearity remains. It is the PC or FA scores that are independent..but they are a function of the whole set of variables. AT this step I would work with the FA as opposed to PC scores because they are usually easier to interpret (i.e. variables load almost exclusively on a single factor as opposed to loading on multiple principal components.). So you would have three "variables" in the regression, each being one of the factors. Your regression is now a linear combination of the Factors, which are in turn linear combinations of the original variables. Working through these two linear relationships provides a linear function of the yield as a function of the original variables. The

path coefficients provide an index to the independent contribution of groups of inter-related variables in each of the three factors. Each factor represents the joint contribution of the underlying variables that unfortunately cannot be uniquely partitioned due to the co-linearity.

For prediction of yield , you cannot predict the effect of change in an individual variable, but rather must settle for the change in yield associated with settings of groups of variables that are consistent with the combinations in the original data.

I'm attaching an example that is way outside your area of application, but discusses the same issues...see what you think

That is quite in-depth explanation of the scenario.

I have used principal component regression in one of my paper "Genetic Divergence in Sugarcane Genotypes" published in American Journal of Experimental Agriculture Voume 3(1), 2013. Here the principal components went into regression with yield and explained the percent of variation by each PC.

However, here, I have to find out which variables have high/positive direct effect on yield so that I can develop generalized selection indices giving a good genetic advance.

Mr. Gokmen analyzed my data for collinearity and he's of the view that some of the variables (having VIF greater than 10) should be excluded from the path model. It means that we further reduced our variables from 9 to 4. These don't have multicollinearity now (I have to check them though).

"This is where taking the additional step to factor scores in the regression model will help interpretation of variables related to yield. Variables with positive factor loadings and positive regression coefficients on the corresponding factor are positively related. Likewise negative loadings and negative regression coefficients are also positively related to yield.

Unfortunately because of the co-linearity you cannot distinguish which individual variables are the unique drivers of yield. Results should be qualified that yield is positively influenced by certain factors, but that the factors themselves represent information from a combination of primary variables that cannot be parsed individual from your experimental data alone.

Followup work would be designed to split out the contribution of these individual variables through additional designed experimentation.

Hi Tahir, in your discussion, I observed that u were removing both variables which participate in multicollinearity. But its enough to remove one of them. Also try to replace the both variables by a single variable such as taking average of the two variables etc..,

Yes sir. Actually we used VIF (variance inflation factor) of Ridge regression as basis for removing the variables. And that is a good idea to average the variables into a single variable.

You can use Ridge regression method and Modified Ridge Regression method to solve Multicollinearity Problem by avoid the resulting large sampling variances of the ßs. This URL will be helpful.

http://www.m-hikari.com/ijcms-2011/9-12-2011/rashwanIJCMS9-12-2011.pdf

Thank you Dr Huda!

I used Variance Inflation factor values to overcome the problem. One can do it in SAS very easily. The other indicators are the condition index and tolerance. Removing one of the multicollinear variables rectifies the problem.

However, could you kindly guide me as to how to winsorize data which has many outliers? I have a data which gives non-significant F-test for different variables due to this problem. Thanks.

There are may ways to deal with multi-colinearity, you need to wisely choose the best option.

Dropping some variables (with high VIF) or getting more data is not always suggested.

For variable selection, automated processes (like step-wise method) should not be used, as stepwise regressions are controversial and might lead to model misspecification. [Paper by Peter Flom: http://www.lexjansen.com/pnwsug/2008/DavidCassell-StoppingStepwise.pdf]

 You can use PCA to reduce dimensions of the model or use can use regularization/ penalization (L1/L2 norm or combination of both) or least angle regression (LAR). I personally prefer using ridge/lasso or elastic net for dealing with the issue.

If you are using R for your analysis, it's easy to tune the parameters of the model by using cross-validation.