Exactly the same question was asked a few days ago. David Booth and I both responded. However, I am a new to RG, so I there are a few things I don't know how to do. One is to refer to another Q&A thread. If any one wants to educate a newbie, I would appreciate it. Since I don't know how to redirect, I am forced to be redundant and post a quote from the other Q. Here 'tis. If you need more, let me know, and Best of Luck in your research

===============

This may be perceived as a relatively simple question, or one that is far more complex. Let’s first assume that you can transform data such that a linear relationship (if any) exists among variables, so that regression methods apply. Note that this does not mean that simple functional relationships that are expressible in a linear form are a problem (e.g., a polynomial model).

Here is a simple answer. A ridge regression is achieved by adding a small constant to the diagonal of the predictor variable covariance or correlation matrix (depending on how you scale) before inversion. This could be one constant (referred to in ridge’s introduction by Hoerl and Kennard as “k”) added to all variables’ diagonal entries, or a different constant for each variable (this is often referred to a generalized ridge). The original method of Hoerl and Kennard was one of increasing k from zero (OLS solution) in small increments to a “stopping” value (often 1.0 was used). Then subjectively considering the increase in stability of the regression weights as k increases. Then, selecting a k that offers satisfactory stability. This plot is called the Ridge Trace. Hundreds, if not thousands, of publications have posited theoretical as well as empirically derived alternative methods of determining k – far too much to consider here, although some of this is in references I provide.

This decision is, however, a compromise, as k increases, stability increases (variance of the weights decrease, leading Richard Darlington (1978) to deem this, and some other methods, Reduced Variance Regression), yet as the OLS weights are BLUE, the RSQ decreases from its maximum as k increases. So, this is probably best considered as a trade-off decision. For instance, SPSS yields the aforementioned Ridge Trace (illustrating increased stability), but also a plot of the monotonically decreasing RSQ as a function of k (illustrating loss in accuracy in the sample at hand). From this information, you may be better able to make a compromise choice of k. Note however, that it is in cross-validation that this and other alternatives to OLS can excel in accuracy over that available from OLS.

Although ridge regression is available in a variety of statistical packages, if it is not available to you, Darlington cleverly showed how it can be obtained with a standard regression package using a “shrunken” intercorrelation matrix. I would caution however, that in certain circumstances (e.g. seeking a stepwise solution) results can be incorrect; see this, if interested:

Article Stepwise ridge regression: A computational clarification

[Gee, I thought that this was simple before I started writing it😊]

You mention multicollinearity, and this is often the springboard for ridge regression. However, the case is that multicollinearity is only a necessary, but not sufficient, condition for ridge regression to exceed the cross-validation accuracy of OLS. Validity concentration (a term coined by Darlington in the same article) is also necessary. Roughly, this regards the degree to which predictive accuracy is associated with large PC eigenvalues of the predictor variables' R matrix. Thus, collinearity is necessary to allow large eigenvalues at all, but given large eigenvalues, validity concentration may, or may not, be the case depending on the correlation of the criterion with these PCs. If you are interested, we have attempted to corral the notion of validity concentration formulaically, and to validate that formula here:

Article Validity concentration formula validation

There is so much more to say, particularly regarding a caution of the potential loss in accuracy that may be afforded by ridge regression if the aforementioned conditions (and others) do not obtain. Rather than try to tackle that here, I will refer you to a piece that includes much of what I might say:

Article Ridge regression and some alternative weighting techniques: ...

In addition, there have been many studies that further examine the cross-validation of ridge and other alternatives to OLS. You may be interested in one of my project folders that I am currently building with some of my results:

https://www.researchgate.net/project/Ridge-Regression

Hi You can find the previous discussion that John Morris alluded to at this link:

https://www.researchgate.net/post/If_I_have_data_distributed_by_Weibull_and_the_multiple_regression_model_with_Mutlicollinearity_how_can_I_use_the_Ridge_regression_method

Best Wishes, D. Booth

Thanks David.

Interesting topic but it is away from my field

Interesting topic

Interesting