You are making an overparameterized model, however it is better to made a model simpler as much as possible and you could drop one or more variables from the analysis.

Dear Chidi,

I think you could try two things:

First, as Robert suggests as well, drop the variables which show the redundancy. For instance, just compute a correlation matrix between all variables, and then retain one in each pair of variables, that highly correlated.

Second, and I think, this "complicates" things: the message you quote indicates that you are using a 'principal component' analyses (PCA). I do not want to be prejudiced :) but researchers usually are not aware, that this is doing something else than a 'principal factor' analysis (PFA). Simple example: 10 questions on an online survey, each question on a separate page/slide, and you measure the response times. Now assume, the 10 questions will be differently long, so - despite the text length - you would probably agree that there is most probably no 'latent factor' behind them. Nonetheless, when you conduct a PCA on the response times, it will give you components that 'explain' a large amount of variance. And this simply comes from 'item specific' variance. That is not desirable from a 'latent factor' approach, which tries to identify relations on a deeper level. That is what the PFA was 'invented' for (for anlalyzing intelligence tests, independent from the type of questions used).

Now given that you use a PCA (which you may want to think over), the basic goal of this 'thing' is to generate components until everything is explained. The PFA on the other hand only tries to explain what 'really' correlates with each other. In terms of some simple math (oversimplified): the (almost) only thing that differs between those analyses, is that a PCA uses a (correlation) matrix with diagonal values of 1's (meaning that each item can be fully explained), whereas the PFA uses other values in this diagonal (i.e. some value between 0 and 1, e.g. the variance explained by all other variables as obtained in a corresponding multiple regression, but there are other methods). Anyway, this difference also points to other possible solutions.

1) for PCA, constrain the model, such that the maximum=allowed number of components is very low. This will have the result, that the data is not fully 'explained', and is, by the way, a common and desired implementation. Since the goal of the PCA is variable 'reduction'.

2) Use a PFA, if the variables you have are actually -not- highly redundant in a correlation matrix (where a PCA will still find shared item specific variance, which you do not want). If the PFA still gives you the same error message, then the rough explanation above gives you a hint why: because one (or more) variables are linear transformations of two (or more) other variables, and therefore can be obtained like, e.g. variable 1 = weight1*variable 2 + weight3* variable 3. In this case variable 1 can always be fully determined by both other variables, which, in short, makes your model unidentifiable (i.e. the estimation of the variable weights, just like in regression, but there you speak of 'tolerance'). Actually, I can not come up with many situations in which this should be the case, other than: You have measured a variable but systematically manipulated some of the other variables, which therefore do not reflect measurements but deterministic item-characteristics (Please, do not run a PCA or a PFA over this :)). If this pattern still evolves with empirical measurements, then drop the co-dependent variables. You can run multiple regressions from each variable, on all of the other variables. If the variance explained by all of the other variables is near 1 for the predicted variable, then simply drop the predicted variable :). (And do not hesitate, because you do not lose any information when dropping this variable, since it can be fully explained by others.)

Hope this helps :)

Best, René

Rene gives some really good advice.

One of the causes of exact collinearity so that the determinant is zero and the matrix cannot be inverted is that there is a closed number set in you variables. To take the simplest example say when there is two proportions ( or percentages) that sum to 1 (or a 100) across the row. E.G. % Male and % female - this will always have a correlation of -1; the solution is then to drop one of them - the other is redundant you already have the information. The same thing happens when there are m categories forming a closed number set - you have to drop one as the m-1 variables contain the all the information there is. (The correlations between pairs of variables may not be high, eg 5 category set with each variable having approxirately the same variance, the expected correlation is -1/(m-1) = -0.25 but the determinant will be zero.)You can also get a similar problem if a variables does not vary - it is constant.

The advice - before doing something more technically demanding - make sure you variables vary (look at the variance) and conceptually you do not have a closed number set.

The large extract you give is talking about a ridge procedure where you get the correlation matrix of the variables of interest and the diagonal will be a set of 1's - the correlation of a variable with itself - you then add a small number say 0.01 to each and every value of 1 on the diagonal and invert that matrix; it is a technical fix to invert the matrix.