Usual practice is to report the analysis with all dummy codes - not significant or not. If it helps you could switch the coding round and use the significant dummy code as the zero code. This would produce the same pattern but possibly both coefficients would be statistically significant (though this is not guaranteed). I can't see any value to dropping the non-significant coefficient if you had a reason to include in the model a priori.

Hi Chea,

You can perfectly gather levels of a categorial variable. In your example, this is equivalent to exclude the dummy variable for the non-significant level. Check the other coefficients of your model to verify whether there is a significant impact of excluding this dummy of your model. If not, it is quite okay to remove it.

(Note that this remark does not apply for intercept. With the exception of very specific cases, intercept should never be removed from a model, even when not significant).

I hope this helps,

Best regards

Dear Chea,

Try to check the significant effect of the interaction between the two factors. If there is a significant difference, do not exclude the interaction effect. and generally speaking, you can exclude the dummy variable with non-significant level.

Regards,

Mahmoud

Usual practice is to report the analysis with all dummy codes - not significant or not. If it helps you could switch the coding round and use the significant dummy code as the zero code. This would produce the same pattern but possibly both coefficients would be statistically significant (though this is not guaranteed). I can't see any value to dropping the non-significant coefficient if you had a reason to include in the model a priori.

There is no rule that non-significant variables should be excluded from the model. Further in your case it is part of a variable.

There is no rule that you have to keep non-significant variables in the model. There is a good argument for keeping the model hierarchical. That is, if the ABC interaction is significant then you keep the AC interaction term even if it is non significant. If the ABC interaction is not significant then you can remove the AC interaction if it is not significant.  The reason for removing a non-significant variable if it was included in the model a priori is that your original concept for the model was flawed.

I would suggest checking the residuals. Look for patterns in a scatter plot, do a Q-Q plot. That may help.

To me, dropping an insignificant dummy variable doesn’t make much sense. 

Imagine the case where you are measuring the amount of sugar in three colors of apples: red, green, and yellow.  Let’s say red is the base level, so you have two dummy variables:  one for green and one for yellow.  Let’s say the green is significant and the yellow is not.  If you want to drop the yellow variable, what do you do?  Have red and yellow as the base and one dummy variable for green?  That seems different than what you wanted to test in the first place.

More thoughts:

From a practical point of view, I find regression results listing dummy variables to be undesirable.  It seems to me that the question you want to answer is, “Is there an effect of color on sugar?” Not, “Is there an effect of green apples relative to red apples on sugar?   And how about yellow apples relative to red?”.  The solution is to generate an anova-like table, where there is a single p-value for the main effect of color.  Statistically this can be done by testing a hypothesis of the dummy variables together with contrasts.  Luckily, most software packages are able to do this for you at least for some types of models.

@Salvatore

If i understood the example in the first paragraph then I am really testing two different univariate models. Model 1: is red different than green? Model 2: is red different than yellow? In the first case the answer is yes. In the second case the outcome was inconclusive. You are not saying that yellow=red, so combining them would not make much sense. I suppose that you could raise the level of significance such that a p-value of 0.4 is deemed significant. Mostly I am not a fan of that approach. Sometimes experiments fail, or don't go as planned. Now you get to figure out why. Maybe yellow apples have twice the variance of green, and green has the same variance as red. Maybe the distribution of sugar in yellow apples is highly skewed. Maybe the distribution of sugar levels in yellow apples is uniform over the range in values, while the distribution in red apples in Gaussian. What if sugar levels in green apples are correlated with levels in yellow apples, yet yellow apples are still not significantly different from red. If both variables stay in the model, the relationship between red and green becomes less clear.

Maybe sample size is so small that you are drawing awesome conclusions from random noise.

@Timothy,

A clarification:  Not two different models.  It's one model, with multiple terms, and a hypothesis being tested for each term.

@Salvatore

If you have R = G + Y (plus constant and error terms) and you find that R = 4G + 7Y you then need to make a model R=G and R=Y. If R=4G and R=7Y then we can conclude that G and Y are independent and the difference in multiple regression versus univariate is mute. If R=3G and R=6Y then there is an interaction term that is missing and significant in the original model. Alternately, G and Y are correlated. Either way, you have violated the rules of the game. Keep the model hierarchical, and satisfy all the assumptions for GLM models.

@Timothy,

To be frank, I have no idea what you are talking about or why you are arguing with _me_.

The question was about the case where there is a dependent variable and a categorical independent variable that is coded in the model with dummy variables.  In light of that, I can't make heads or tails about what you are trying to say.

@Salvatore,

You stated "Not two different models.  It's one model, with multiple terms..."

I replied that there is no difference between the two if there is no interaction term and the independent variables are independent of each other. So maybe I didn't quite understand your point in the quoted text.

I was not clear. R=red apple, G=green apple, Y=yellow apple, so this is using your apple example -- or at least trying to. It may also be that I did not understand when you stated "have red and yellow as the base and one dummy variable for green?" I interpreted this to mean that you could combine the red and yellow into a single category with red and yellow mixed and then contrast that with green.

It doesn't make any sense to remove variables just because they're not significant. In some cases significance might be a useful piece of information if you have other reasons to remove variables (e.g., if you need to simplify a model that won't converge, or if you're trying to make a "parsimonious" model with as few terms as possible). But as far as I know there's never any point in removing a non-significant variable just because it's non-significant; there's always some other reason. So if you don't have any a priori reason to want to exclude variables, you don't need to even worry about this; just make a model with the variables you had a priori reason to include.

Usually it is best to have a hierarchical and parsimonious model.

What I think is reasonable might not be what you think is reasonable. So I like The Club (Hong Kong football team), and I think that the number of times The Club wins this year will influence corn yields. Since it was entered a priori it should stay in the model?

Another approach .... you and all the other scientists agree that nitrogen improves corn yield, and yet you run an experiment that shows nitrogen is not significant. There are a number of possibilities:

1) Your experiment was too poorly executed to detect the effect.

2) Your replication was too small relative to the expected variance to detect the effect.

3) There is already sufficient nitrogen in the system, adding more has no effect

4) The nitrogen was contaminated with some toxin that retarded growth to the extent that added nitrogen would boost growth.

5) The paradigm is wrong. Experimental methods often drive experimental outcomes, and you have finally found a method that shows a difference.

6) Something not listed above.

If a variable is really important, then the finding that it was not significant may simply be telling you to take a step back and rerun the entire experiment after rethinking the methodology.

If everyone agrees that the variable is a critical element of the model, and you show that it is not significant, then I must assume that the reason is one of the first four in the above list. Why should I not reject the manuscript at this point?

If you need another reason for removing variables try looking at things like AIC, BIC, or adjusted R-square. Please note that you should always state the entire list of variables that were examined, and state that "these were not significant and therefore dropped from the final analysis." Some people might want to know the significance level for dropping a variable as they might feel differently about a variable dropped with a p-value of 0.0556 versus 0.789.