ISSUE: How to choose control variable Z in partial correlation rxyz?

MULTIPLE REGRESSION: The issue of partial correlation may arise when one engages multiple regression modeling. Assume that there are three variables X, Y and Z. All three variables are random. The correlation matrix of X, Y, and Z may be laid out below:

          X     Y     Z

X       +

Y        rxy   +

Z        rxz   ryz   +

The partial correlation of XYZ is:

(1)   rxzy= (rxy – rxzryz) / sqrt (1 – r2xz)(1 - r2yz)

The test statistic is given by;

(2)   txyz = (rxyz(sqrt(n – 2 – k) / (1 – r2xyz)) tn-2-k

… where k = number of variables in the model; and n = sample size.

HOW TO SELECT Z: The same rule for selecting X and Y. The variable must be random and IID. In the several Xs that you are working with, there might be another variable that exerts influence on the correlation between X and Y---this is the suspected Z. How do you select Z? The answer is “what is your suspect?, i.e. which variable do you suspect to exert influence on the correlation between X and Y among your various Xi: x1, x2, …, xn)---that alegedly inflates the correlation coefficient between X and y? The idea is to remove that inflation.

IF Z HAS RELATIONSHIP WITH Y BUT NOT X: If this is the case, the issue of partial correlation becomes moot. The correlation between X and Y, in that case, is solely contributed by X only---no Z. If Z has a relationship with Y and not with X, this means that you have an incomplete model. There is another variable called Z and can explain Y. Recall you multiple regression model is given as:

(3)   Y = B0 + B1X1 + B2X2 + ... + BnXn + e

In this case, Z could be Xn.

REFERENCES:

(1) Baba, Kunihiro; Ritei Shibata & Masaaki Sibuya (2004). "Partial correlation and conditional correlation as measures of conditional independence". Australian and New Zealand Journal of Statistics 46 (4): 657–664. doi:10.1111/j.1467-842X.2004.00360.x

(2) Kendall MG, Stuart A. (1973) The Advanced Theory of Statistics, Volume 2 (3rd Edition), ISBN 0-85264-215-6, Section 27.22

Thank you very much for your kind help, Mr. Paul!  I have a new question. I got some independent variables, X1, X2, X3, of dependent variable Y  by stepwise regression method, which also provide the partial correlation coefficient between each Xi and Y. I found that the partial correlation coefficient between each Xi and Y is the calculated with other independent variables as control. But, actually, X3 dose not have significant linear relationship with X1 and X2. So, dose this partial correlation coefficient make any sense? Or I have a wrong understanding? Thank you very much!

This is a tough question and the applicability of any answer will depend on the particular theories/hypotheses that you are interested in.  There is a large literature on causal models (e.g., Pearl, 2009, Causality, but to get a feel for his approach see his wonderfully entertaining talk http://singapore.cs.ucla.edu/LECTURE/lecture_sec1.htm, which is reprinted in the 2009 book). 

As far as "stepwise" approaches to variable selection in regression, most methodologists are against the standard methods.  For automated methods many advocate using methods based on the lasso (Tibshirani, 1996; see http://statweb.stanford.edu/~tibs/ElemStatLearn/ for coverage of it in the [is it old enough yet to be] classic textbook of Hastie et al.), but as above it is often better to base your choice on what you know about the variables.

Yes, X3 might be important if you consider the semi-partial correlations. Because even if X3 does not correlate with X1 and X2 if it shares some Variance with Y it holds a piece of Variance constant and thus X1 and X2 have a greater percental impact on Y (or on the Variance of Y - X3).

It's a question of theoretical justification whether you want to point out how much your independent variables correlate with the whole variable Y (partial-correlation) or with the part of Y Variance you don't know the origin of (semi-partial correlation). 

The picture attached shows a regression analysis with three variables X1, X2, X3 and it's correlations (null, partial, semi).

-The change from Null to partial correlations shows how much your variables are distinct from each other. The bigger the change, the bigger the correlation between the factors (Xi).

-The change from Partial to Semi shows how much Variance of Y is shared with other factors.

So: If a factor loses much of it's correlation to Y from Null-->Partial-->Semi-partial then it means it is similar to other factors. If it stays robust then it shares Variance with Y that is not explained by other factors.

Yes, it does make sense. X3 may be a significant factor. Just because it exerts no linear relationship with X1 and X2, it does not mean there is a problem. Now test for statistical significance of X3 on Y. If there is significance then X3 stays as part of your explanatory factor.

Cheers.

Thank you all for answering my question! I think it is better to perform partial correlation when control variable Z has significant relationships with both Y (dependent variable) and X (independent variable). Under this condition, we can get a relative pure correlation between Y and X, which with the effect of Z removed. But, actually,  if Z has no significant correlation with X and Y, the partial correlation coefficient also will be changed. But I think this change in value will be small and  meaningless.  Do you think so?

Yes I think so too but failing  the significance test on your correlations does not render them meaningless. It's up to your argumentation how high or low your correlations are considered important for your study. If you do a regression analysis then adding a factor might raise your R² significantly but the correlations might be not significant with the new factor..