Nabin, this may be a suppressor effect. This effect may occur when two predictors are both positively correlated with the criterion variable but negatively  correlated with each other. 

Regards,  Karin 

Karin, Thank you for your prompt response!

Yes, that appears to be the case. I have not dealt with the suppressor effect yet. Would you please point some relevant papers on the subject. I would like to learn about how to examine further the suppressor effect, its implications and how to report it in the paper. Many thanks for your help.

Nabin

Hey Nabin,

regression coefficients are partial correlations so they are different to a correlation between two variabes. The partial correlation is the correlation between two variables with the effect of your control variables is removed.

There can be further reasons why the sign of the estimated coefficients contradicts your expectations. Perhaps you have outliers in your sample or you have forgotten important regressors.

Best regards,

Florian

Nabin,

here are some articles, but I think that you would find this problem under "reciprocal suppression" in some textbooks, too:

Conger, A. J. (1974). A revised definition of suppressor variables: A guide to their identification and interpretation. Educational and Psychological Measurement, 34, 35-46.

Tzelgov, J. & Henik, A. (1991). Suppression situations in psychological research: Definitions, implications, and applications. Psychological Bulletin, 109(3), 524-536.

Massen, G.H. & Bakker, A.B. (2001). Suppressor Variables in Path Models: Definitions and Interpretations. Sociological Methods & Research, 30(2), 241-270.

Kind regards, Karin

Hi Nabin,  This sounds like suppression.  Here is an article in which we had a similar effect and our effort to explore and explain it.  One way to explore why suppression occurs can involve similar processes of searching for mediation/moderation. Taking the variable in question in exploration iterative (leave one out) regressions to determine interactions between predictor variables. 

Stern, M.J., and S.A. Predmore 2012. The importance of team functioning to natural resource planning outcomes. Journal of Environmental Management 106: 30-39.

Good luck!

Marc

Thank you Florian, Karin and Mark for the suggestion and relevant papers. After reading those papers, hopefully I will be able to solve the problem.

Nabin

What is the VIF/tolerance among the different terms in your model?

I've come across this issue a couple different times. There are usually 2 issues that bring it up.

1) The VIF of certain terms in the model is high.

2) The original regression analysis was poorly performed. (This is a big issue in a lot of the science articles I read for my classes.) 

Both issues have the same root cause, confounding. This occurs when you have a response variable that is function of many variables but few are selected or a function of a few variables and many are selected. 

You might find some helpful links by looking for something on Simpson's paradox. 

Thank you Andrew. The VIF appears to be reasonable. One variable has the value of 6 but the average VIF is around 3. I think I read somewhere that multicollniearity is a real issue only when VIF is greater than 10.

Nabin

Hey Nabin,

Some say 5.00. Some say 10.00.

What happens to your model if you remove the term/variable with a VIF of 6.00? 

Andrew: If I remove that specific variable then there is no problem. But that variable is critical to answer the research question so I don't want to exclude it from the model.

Dear Nabin,

What kind of software did you use?

Because there are issues on computing the generalized Moon-Penrose matrix.

If you don't want the betas and only the fitted values you can use the method described on my publication.

Article Linear Regression without computing pseudo-inverse matrix

According to me, you have a problem of multicollinearity. The VIF test is a good warning. use the stepwise regression. You can also check the presence of outliers and then retry the analysis after their removal.

You have to be careful. a relation between any two variables is said to be spurious relation. The total relation is made of Direct and indirect effects.

Beta on the other hand measures the true direct effect.   therefore if the total spurious effect is positive and the true  effect is negative then Beta is going to be negative. 

That is why the simple relation coefficient goes from -1  to +1  while B weights and Beta weights in standardized score may take any value (more or less than 1.0) 

It might be indication of presence of multicollinearity and /or interaction effect. Read more:

http://www-bcf.usc.edu/~gareth/ISL/  

When we partial the effect of one or more independent variables we may end with a partial regression coefficient having an algebraic sign that is different from the simpe correlation coefficient

Dear Nabin, everyone,

I have faced the exact similar issue in my analysis. I am sharing the link to my question. As indicated in my question, no sign of significant VIF was observed (the max value of VIF in my independent variables is 1.4). However, the coefficient changes its sign. I was wondering if could solve your issue Nabin? Or any other thoughts would be greatly appreciated. 

https://www.researchgate.net/post/Why_does_an_independent_variables_coefficient_change_its_sign_in_regression_model_when_it_is_input_along_with_other_predictors

There are a few remedies to deal with multicollinearity. Disturbance is sometimes helpful while dealing with multicollinearity. Also helpful may be keeping the number of degrees of freedom minimum.

The recursive least squares algorithm (RLS) is the recursive application of the well-known least squares (LS) regression algorithm, so that each new data point is taken in account to modify (correct) a previous estimate of the parameters from some linear (or linearized) correlation thought to model the observed system. The method allows for the dynamical application of LS to time series acquired in real-time. As with LS, there may be several correlation equations with the corresponding set of dependent (observed) variables. For the recursive least squares algorithm with forgetting factor (RLS-FF), acquired data is weighted according to its age, with increased weight given to the most recent data.

Years ago, while investigating adaptive control and energetic optimization of aerobic fermenters, I have applied the RLS-FF algorithm to estimate the parameters from the KLa correlation, used to predict the O2 gas-liquid mass-transfer, hence giving increased weight to most recent data. Estimates were improved by imposing sinusoidal disturbance to air flow and agitation speed (manipulated variables). The power dissipated by agitation was accessed by a torque meter (pilot plant). The proposed (adaptive) control algorithm compared favourably with PID. Simulations assessed the effect of numerically generated white Gaussian noise (2-sigma truncated) and of first order delay. This investigation was reported at (MSc Thesis):

Thesis Controlo do Oxigénio Dissolvido em Fermentadores para Minimi...