Attached are the results of my regression with collinearity statistics.

Vladimir -

I am not familiar with your terminology  (e.g., "Theory of Basic Values"), but if you are saying that some small level of collinearity is common, I think that sounds reasonable. How much is OK is a problem perhaps answerable by examining model outputs.  You might even compare model performances by comparing their estimated variances of prediction errors.  If you have data you can use for test data, then cross-validation would also consider bias. 

If you can show good test results then that may satisfy your supervisor.

Cheers - Jim

Dear Jim, 

Thank you very much for your time and your feedback! 

And sorry, the psychological theory here is not very important. Basically what I have is 1 DV and a set of 10 predictors, all interrelated. I am attaching a file with some illustrations. 

My dependent variable is correlated with all of the predictors in a way consistent with their pattern of intercorrelations. Now the psychological theory says that this is because all 10 IV's have a causal relationship with the DV.

I have a hunch that it is not true, and that there are cases where only one of the IV's has a causal relationship, and the correlations with other IV's are due to the structure of their interrelations. 

I use a simple regression on a very large dataset and find that some IV's give no weights. I come with this to my supervisor and he says this means nothing because there is a multicollinearity problem. I show VIF and tolerance and he says this means nothing because there is a multicollinearity problem. It is very frustrating. 

I read up on multicollinearity problems and find descriptions of cases where high VIF can be ignored. I do not find much on cases where low VIF can still indicate problems. Could there be theoretical limitations as to how to interpret the regression coefficients with non-1 VIF (apart from them being inflated proportionately to VIF)? 

I also understand that regressions are not experiments, I use this analysis to derive hypotheses for the experiments. Which my supervisor obviously doesn't support.

I use these regression weights on a different dataset, and find that they interact with individual-level IV's and add a sizeable amount of variance to the predictions. Again, I am told this means nothing. 

I am either missing something, or my supervisor does. Since he is a very established expert on stats in my field, I am very willing to consider the former. 

This is very frustrating. Attaching our correspondence as well. 

Sorry.  I have never heard of the term "patterns in the predictors," but then statisticians don't look into causality like econometricians and perhaps others do, and it sounds like terminology designed to deal with that, maybe.  Or maybe I just never ran across it and other statisticians have. 

There are a lot of papers loaded on ResearchGate. Perhaps you could search them (top of your profile page in the search box, change the person icon to publications and search there), and when/if you find something that seems on topic, if it does not help enough, you might follow and then write to an author to see if you can get a useful explanation.  Papers alone might help though, and give you references to use.

Vladimir -  Just guessing, but with 10 regressors, though most look good, could your supervisor be saying he/she thinks that model is an overfit?  That is, if you used all your data to select the model, it may be an overfit to those data, and may not generalize well to the population.  (If you split your data and use part to get a model, and the rest to test it, that might help.)  There is a bias-variance tradeoff, and more (non-collinear-problematic) regressors can reduce variance but increase bias.  Could that be what he/she is talking about?  Perhaps he/she thinks you should concentrate on fewer regressors??? -  Just guessing.  It is often hard to know what someone else is thinking.   Perhaps your supervisor is not particularly good at expressing hlmself/herself.  That is not unusual. - Best wishes - Jim 

PS -  By the way, my big interest is in heteroscedasticity.  If you have scatterplotted the estimated residuals, perhaps you may have found that you have this often naturally occurring phenomenon.  (That is what I would call a "pattern.")  But I doubt that your supervisor is paying attention to that.  ---   Anyway, perhaps the concern your supervisor has, as noted above, can be researched under the term "overfitting," but, as noted, that is only my guess. 

One other concern: The intercept is about 2.4 with a relatively small estimated standard error. If 2.4 is not a small part of predicted y for most of your y-value range, then there should be some good explanation for it. Because if it is really a result of other regressors that are left out, and you have a number of regressors already included, then you might not only want to reduce the total number of regressors you use, but also replace some of them with others. But again, I am just guessing.  (I have not worked with such a case.)  You might want to reconsider the subject matter theory behind your modeling to see if you want to try something else - a different set of regressors.  (You may be restricted greatly, however, by what data may be available or feasibly collectable.)   But again, this may not be what your supervisor has in mind at all. 

Vladimir -

Hmmmm.  According to Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning, Springer, 2009, second ed., page 221, "As a model becomes more ... complex, it uses the training data more and ... there is a decrease in bias but an increase in variance." 

Well, I can see it decreasing bias conditional on the training set, but since that can overfit the training set, that sounds like increased unconditional bias to me, but perhaps I got it wrong.  Sorry.

Jim

Dear Jim,

Thank you very much for your time, knowledge, and thinking that went into these replies! This is very, very helpful. 

I will try to describe what I have done about different suggestions you made.

Overfit.

Here are the results for the same regression on another large dataset (same social survey, 50000 participants, collected couple of years earlier than the first one). As I am only interested in very rough estimates of the weights, basically the size and direction, I was very happy to see these numbers. I have put the unstandardized coefficients from the first regression into brackets:

High intercept.

This is a field thing. In pyschology we generally don't explain all that much variance, and the theory I am working with is especially lacking in predictive power. I am not trying to predict as much variance in the DV as possible, my main interest here is the structure of relationships between these 10 particular IV's and the DV. 

I will try to explain what I am trying to do with these regressions with an analogy. I do not know how successful it will be. 

Let's imagine that I am interested in the effects of monthly temperatures on average indoor temperature in different cities of the Northern hemisphere. The theory argues that all monthly temperatures affect the indoor temperature. I want to argue that it is not only the monthly outdoor temperature that affects indoor temperature, but also the schedule of public heating in these cities. 

The data that I have are the monthly temperatures for the cities from different years, and the average yearly indoor temperatures in these cities. 

The data that I do not have (but am very interested in) is the heating schedule. 

I run a regression with monthly temperatures as predictors of yearly indoor temperature, and find that some winter months give no weights. I try to argue that during these months, the heating is on, and go on to test it in the experiments.

In the meantime, I use the regression weights in an analysis of a new dataset, and see that they interact significantly with monthly temperatures, to improve my prediction of yearly indoor temperature. 

After arguing with my supervisor for some time, I think our disagreement might lie in the field of psychological theory, and not in statistics (although he does not see it that way).

What he keeps repeating to me is that I cannot look at regression weights because they ignore the interrelatedness of monthly temperatures. But in this particular scenario, I am pretty confident this is exactly what I want to be doing. 

Vladimir -

Well, if your supervisor does not want you to use regression, although it could be used to check out various relationships, it would take some really good results to convince him/her otherwise, so you might have to try something else for now, if you have any idea what might satisfy. 

I'm sure I did not follow all of this - the application, and also the use you made of the term "regression weights," which is used for expressing the heteroscedasticity in weighted least squares regression, but I don't think that was how you used it - so this may again be off topic, but you might be interested in this:

 https://www.researchgate.net/publication/48178170_To_Explain_or_to_Predict

Oh, it occurs to me that you might have been saying that you were looking at regression coefficients, and because there are interactions between regressors, your supervisor says they are not useful.  Well, they are changed.  If you want to look at an individual regressor, in 'simple regression,' its coefficient will generally be quite different from when it was part of a multiple regression.

At any rate, I hope this gets to be a lot more fun for you soon. 

Best wishes - Jim

Article To Explain or to Predict?

Heteroscedasticity.

Here are the scatterplots of the residuals. I'm not sure I've done it right, but if I did, I guess this suggests a certain degree of heteroscedasticity that is still unlikely to significantly affect my SE estimates.

My knowledge is very inadequate here, so I might be making very silly mistakes. 

In any case, thank you again for your time, this all helps me a lot to learn.