Dear Homa,

Thanks for sharing your interesting question. I suggest that you're sharing nothing but the tip of an iceberg that holds more insights about your data and about regression. As someone who thinks about exploratory data analysis quite a bit--and how to do it well without making it a purely opportunistic crapshoot, I have taken the occasion of your question to compile a few remarks.

The short version is that I suspect you are dealing with the correlation between predictors, and I think there's no problem unless three conditions hold (population is having this effect on the other predictor AND the other predictor remains significant AND there is no significant change due to population). Suppression is a known feature of regression modeling, but the latter three-condition problem would seem like overfitting. I don't know enough about your dataset to diagnose further.

Longer comments below:

You have found a case in which linear regression shows itself to be a bit more lively and globally sensitive than we typically think linear regression should be. Your dismay reflects the absolutely sensible impression that each of the coefficients in a regression model reflects an independent weight of a component predictor in a grand sum meant to add up to the dependent outcome measure. So, you are completely in your rights to feel betrayed and bewildered.

The rarely documented or discussed aspect here is that you, the modeler have an interpretive view here that is not linear and not dependent. So, as we run regression models, we put ourselves in the position of an artist trying to paint the most judicious picture. Classically, statisticians and scientists will paraphrase Picasso to say that "models are a lie that show a little bit more of the truth." So what you are expressing in this question is nothing different from when the artist takes another step to the left and may see that something in the visual field that seems one color takes a different shade in a new light or from a different perspective.

The main reason that the inclusion of a new predictor should change the estimated coefficients of predictors in smaller/different models is that your new predictor is in some way correlated with your earlier predictor.

And crucially, the change you see amongst independent coefficents from one model to the next has much to do with the relative change in the model as you include new predictors. The older and usually smaller model is a condition that changes how you might view the effect of the added predictor.

What you might be seeing in your example: In a class of effects in regression called "suppression," it is entirely possible for two mutually positively correlated predictors to return significant coefficients with opposite sign. That is upsetting and strange at first glance because, if Predictor 1 and Predictor 2 are both positive correlated with each other, then they should each also have similar direction of relationship with the the dependent variable.

Why what seems nonsensical may be statistically inevitable due to correlated predictors: The problem is that you are running up against the difference between your awareness of more complex causal relationships than the current math of your model. And you will get these seemingly absurd answers because one or other other of these predictors alone might give the sensible answer you expect, but both together (and more) require carving up the variance of your dependent variable in a way that has less to do with your theoretically driven intuition and more to do with the matrix algebra given the constraints of your dataset as well as the order of entry of the predictors.

There is always a nonvanishing margin between what you can imagine to be true and what you can model with any dataset: The order-of-entry issue is initially very frustrating because, without awareness of the matrix algebra supporting the linear regression, it makes little sense that entering predictors in different orders should give a different answer (e.g., of course the sum of a linear regression is as commutative as any additive relationship), but it's the cold hard truth of, say, rank-deficient matrices that if you add too many correlated predictors, the software will actually just disregard the last entered predictors.

Important questions for the technical issue of model specification are:

1. Is the new predictor theoretically warranted? (just as Ajit kumar Roy

2. Is the new predictor correlated with your existing predictors? (probably yes: gdp and population are likely correlated)

3. Does the new model with the new predictor have a significant coefficient? (that is, is the new predictor pulling any weight for actually predicting change in the dependent outcome)

4. Does the newer model with the new predictor have a model-wide goodness-of-fit statistic significantly different from the older model? (e.g., in ordinary least-squares regression, the question is whether the change in R-squared equals a significantly large F-statistic. in max-likelihood, the question is whether change in -2*log-likelihood equals a significantly large chi-square statistic)

If answer to Q1 is no, then don't include it, but be prepared for someone to disagree if they have a different perspective and so different theory from you.

If answer to Q2 is yes, then it's going to clutter your previous model, but it might reveal confounds that you need to know about rather than accepting the previous model too soon.

If the answers to Q3 and Q4 is both no, then it's worth asking Q1 again, but there's deep rhetorical value to including nonsignificant effects that other people will think is important. However, if the new predictor is nonsignificant AND you also change the sign of the other still-significant coefficient, then you may be suffering the problem of overfitting. At that point, it is worth considering whether you have enough power for the contrast you're trying to make and whether you may be overfitting your data.

It might seem like Question 3 and Question 4 are the same, but they're not because the new predictor may not earn its own significant coefficient, but that's not to say the model as a whole cannot improve significantly. Here is a place where the individual regressions are definitely linear but the process of model building is entirely a nonlinear process blessed/contaminated by the modeler's awareness/motivations/biases.

Ultimately, you are dealing with exploratory data analysis, which I think can be very constructive and informative, but which has a bad reputation for being as slippery as you're finding. It requires the proper couching in your understanding what your theoretical questions are versus what anyone looking at your model is going to want to know. Exploratory data analysis can work quite effectively, but it is hazardous just because these initial flickerings of coefficients with single inclusions or exclusions are not transparent or easy to come to a fast appreciation for. Very often, most of our likely predictors are correlated, not just with the dependent variable but with each other. And that is a problem that you're not going to solve any time soon. So, I recommend getting comfortable with suppression effects or with exploring how to disentangle more orthogonal predictors from your measurements.

Best wishes,

Damian

Hi Homa,

What I will suggest is that if your model is well specified, the model chosen is appropriate taken into consideration the appropriate test for model selection and the variables are properly measured among others, then think about plausible situation(s) that might have led or could lead to the change of the expected sign. You can think of any condition(s)or situation(s) in Iran that could possibly lead to such finding. You just have to justify the outcome; but ensure you have conducted various diagnostic test to ascertain that the estimated resulted are free from any econometric problem.

Good luck and have a nice time

Damian G Kelty-Stephen Dear Damian,

wow! Thank you soooo much for the comprehensive explanation of the issue. It is amazing how you explained the whole possible reasons and aspects! I'm using your tips for my regression and curious how this actually ends.

Best wishes, Homa

Samuel Tawiah Baidoo Dear Samuel,

I just want to be sure the results are significant so that I can interpret them in a right way. Thank you for your comment.

Best wishes, Homa

You are welcome, Homa and all the best.

Regards

Samuel

Damian G Kelty-Stephen Dear Damian,

I ran the both regressions again and got the results:

1) The regression with only gdp values

Fixed-effects (within) regression Number of obs = 8,440

Group variable: exporter Number of groups = 292

R-sq: Obs per group:

within = 0.0915 min = 1

between = 0.3772 avg = 28.9

overall = 0.2432 max = 29

F(2,8146) = 410.42

corr(u_i, Xb) = 0.1861 Prob > F = 0.0000

------------------------------------------------------------------------------

logimport | Coef. Std. Err. t P>|t| [95% Conf. Interval]

-------------+----------------------------------------------------------------

loggdp_imp | 1.357272 .1157355 11.73 0.000 1.130401 1.584144

loggdp_exp | .7134183 .1157683 6.16 0.000 .486483 .9403536

logdist | 0 (omitted)

_cons | -39.25379 1.824615 -21.51 0.000 -42.8305 -35.67707

-------------+----------------------------------------------------------------

sigma_u | 4.1691396

sigma_e | 4.3458386

rho | .47925738 (fraction of variance due to u_i)

------------------------------------------------------------------------------

F test that all u_i=0: F(291, 8146) = 17.50 Prob > F = 0.0000

2) The regression with gdp and the new predictor, population

Fixed-effects (within) regression Number of obs = 8,440

Group variable: exporter Number of groups = 292

R-sq: Obs per group:

within = 0.1277 min = 1

between = 0.1226 avg = 28.9

overall = 0.0885 max = 29

F(4,8144) = 298.01

corr(u_i, Xb) = -0.8304 Prob > F = 0.0000

------------------------------------------------------------------------------

logimport | Coef. Std. Err. t P>|t| [95% Conf. Interval]

-------------+----------------------------------------------------------------

loggdp_imp | .3908133 .1255011 3.11 0.002 .1447991 .6368274

loggdp_exp | -.1544351 .125447 -1.23 0.218 -.4003433 .0914732

logp_imp | 6.528292 .5056391 12.91 0.000 5.53711 7.519474

logp_exp | 4.392624 .5055761 8.69 0.000 3.401566 5.383682

logdist | 0 (omitted)

_cons | -179.8994 7.9409 -22.65 0.000 -195.4656 -164.3332

-------------+----------------------------------------------------------------

sigma_u | 8.8705512

sigma_e | 4.2590406

rho | .81265975 (fraction of variance due to u_i)

------------------------------------------------------------------------------

F test that all u_i=0: F(291, 8144) = 19.03 Prob > F = 0.0000

As you can see, the new predictor is significant and the model also has a higher level of significance. My question is, now that the new predictor is the significant one, and has a more weight on model significance, what should I do with the negative sign of the old predictor (logged_exp). Thank you in advance.

Dear @Homa haddad,

I think that you can take the coefs as they are. The coef with changed sign is no longer significant, indicating only that population may just better predict what your earlier model could only attribute to gdp_exp.

So there's no sign of anything but the collinearity you will have to accept in many multi-measure cases.

If you would like to salvage loggdp_exp, well, you could just pretend the newer model never happened and see how long your argument survives undisturbed without a reader asking the question you yourself did, but I don't recommend that.

Slightly more strategic is to apply principal components to all of your collinear predictors. Then use the principal component scores as predictors to see if the contribution of gdp_exp survives through a principal component relation w one of your other predictors.

Or more easy is just to concede what the second model is saying and elaborate your theory to respect that population may really better explain than gdp_exp.

The elephant in the room here is that vector autoregression might help you explain some of this collinearity, in a way that could include population and without conceding relevance of gdp_exp. Vector autoregression was designed precisely for econometrics. See Sims 1980 and Lutkepohl's New introduction to multiple time series. But that's beyond the scope of the original question. So caveat emptor I guess 😀

Best wishes,

Damian

I suspect that GDP level and population level are strongly correlated in your data (you can check with STATA). If they are strongly correlated then multicolinearity may prevent accurate estimation of their individual coefficients. GDP and population are both measures of size so why use both?

Homa Haddad and Vince Daly ,

It's true that the so-far unclear point of information here is the correlations amongst predictors. However, I think a good reason to include both measures is that, despite the obvious relationship between both predictors (i.e., as Vince points out, they are both operationalizations covering the construct of "size"), there are necessarily a lot of systematic points of nonoverlap between these two measures. For instance, GDP is only size of the economy. It is not size of the work force or even the country. Economic sector, employment rates, and market type are important factors that make for important deviations between GDP and population.

Here we are at the point where point of view of the modeler is an important variable never made explicit in any single model alone. Vince is quite happy for Homa to drop one of the variables, but for me, the genie is out of the bottle, and we see the model telling us significantly different things with the new inclusion.

Who is right? All of us...which is either sad or wonderful depending on your point of view. My feeling is that it's wonderful, but that's just me, one biased participant here.

Best wishes,

Damian