Todd, as fas as I know you are wrong here: ". It is a convention (perhaps a prejudice) in statistics to start with main effects, and to only proceed to testing interactions if the main effects are significant."

The opposite is true: If there is a significant interaction, the interpretation of main effects is not straight-forward. In fact, the main effects can be "non-significant" but there is a strong and relevant interaction. Here is a simple example*,**:

Response: Happiness (let it be measured on a ratio scale; 0=neutral, 0 happy)

Factor A: Gender (female/male)

Factor B: Room color (pink/blue)

Results:

A B response

female pink +2

female blue -2

male pink -2

male blue +2

Here, both main effect (gender as well as color) are 0 (and "non-significant"), but there is a very strong interaction of gender x color (4 arbitrary units of happiness).

Going your way, one would conclude that neither color nor gender are related to happiness, so it would not matter in what color you should paint the room for your daughter or son. But in fact your son would be happier in a blue room, whereas your daughter would be happier in a pink room.

*based on a relatively common (stupid or not) prejustice that gils prefer the color pink over blue and boys prefer blue over pink.

** it is just an example! It may not be sensible beyond its mathematical aspects.

PS: If you have a son and a doughter but only one room, paint it violet... ;-)

Yes, it is certainly possible for parts of a model to be "significant" while the entire model is not. For instance, suppose X explains Y (e.g., "significant" association). It is possible to add to the model (starting with X) parts of no relevance so that ultimately the explanatory part of the model (the X part) is washed out. This because the threshold for the global test of the model is increasing with each additional variable because of increasing degrees of freedom.

Your quandary about main effects and interactions is interesting, but not unusual. It is a convention (perhaps a prejudice) in statistics to start with main effects, and to only proceed to testing interactions if the main effects are significant. Therefore, if results showed no significant main effects, many of us would leave it at that and report null findings. However if there was a compelling reasons to investigate the interactions, we might proceed to look at the interactions anyway. A significant results for one of the interactions can easily be dismissed as type 1 error due to multiple testing, unless it was highly significant (e.g., meets a Bonferonni threshold). If your global test (e.g., test of the main effects and interactions simultaneously) is non-significant, then this is likely a null result.

Todd, as fas as I know you are wrong here: ". It is a convention (perhaps a prejudice) in statistics to start with main effects, and to only proceed to testing interactions if the main effects are significant."

The opposite is true: If there is a significant interaction, the interpretation of main effects is not straight-forward. In fact, the main effects can be "non-significant" but there is a strong and relevant interaction. Here is a simple example*,**:

Response: Happiness (let it be measured on a ratio scale; 0=neutral, 0 happy)

Factor A: Gender (female/male)

Factor B: Room color (pink/blue)

Results:

A B response

female pink +2

female blue -2

male pink -2

male blue +2

Here, both main effect (gender as well as color) are 0 (and "non-significant"), but there is a very strong interaction of gender x color (4 arbitrary units of happiness).

Going your way, one would conclude that neither color nor gender are related to happiness, so it would not matter in what color you should paint the room for your daughter or son. But in fact your son would be happier in a blue room, whereas your daughter would be happier in a pink room.

*based on a relatively common (stupid or not) prejustice that gils prefer the color pink over blue and boys prefer blue over pink.

** it is just an example! It may not be sensible beyond its mathematical aspects.

PS: If you have a son and a doughter but only one room, paint it violet... ;-)

Hi Jochen

I was stating that it is a convention (a prejudice) in statistics to deny the possibility of interactions, if the main effects do not exist. I was not stating that such phenomena are not possible. You give an example of one. This convention is a characteristic of being conservative in statistical inference, or rather, of trying to avoid multiple testing. The convention is to favor simplicity, and main effects are more simple than interactions (which can include not just 2-way but 3-way and greater interactions). In general, the level of evidence needed to prove an interaction, is greater than the level of evidence needed to prove a main effect, unless there is prior knowledge favoring the interaction hypothesis. In some disciplines there is a movement toward interactions in favor of main effects. I have colleagues in genomics who believe we have swiped the genome enough to find all main effects for common disorders, and that the focus should be on interactions of genes.

Todd

Ok, Todd, back to this statement:

"I was stating that it is a convention (a prejudice) in statistics to deny the possibility of interactions, if the main effects do not exist."

It this really the case? Possibly I did not get this correctly, but I find this would be quite a stupid convention...

Additionally, it puzzles me because to my knowledge you start analyzing a model including the interaction. If the data does not provide enough evidence for a relevant interaction effect, you simplify the model by removing the interaction, thereby allowing for a simple interpretation of the main effects. But if there is striking evidence for an interaction, you won't simplify the model and you are left with the more difficult task to interpret the results of this complex model. So, you anyway look *first* at the interaction, and only if you can ignore this, you look for main effects... This is just the other way around as said in your statement.

Jochen

Calling my description of what I have experienced as a convention in the literature as "stupid" is a tad unfair.

This is my experience based on many years of interactions with reviewers and editors. I do not object to having interactions in a model, without main effects, if there is compelling evidence to do so, but in the statistics community such models are met with a higher level of skepticism, because implicitly they indicate a multiplicity of tests were carried out.

I certainly do not object to the "backwards" approach to model building that you describe (as in backwards stepwise regression, and not backwards as in wrong), but at what level of interaction do you start? If you have 10 independent variables under consideration do you start with the 10-way interaction, or should only the 45 two-way interactions be under consideration? These are all judgement calls that depend on the application.

I don't think there is right way and wrong way to doing this, as long as each model building exercise is explained and defended appropriately, with regard to the usual aims of making a model fit well (i.e., explain as much variation as possible, while accounting for over fitting bias) and keeping a model interpretable. I do concede that the second aim of interpretability is not important in some fields, such as in the data mining community (the algorithms approach of computer scientists).

Thanks for the discussion

Todd

Thank you very much Todd for your fast response to my question. Your first answer was very helpful for me, stating that indeed it is possible to have main or interaction effects in MR, although the whole regression model is not significant.

I have mentioned main or interaction effects just as an example but was primarily interested in whether a whole regression model can be non-significant but still resulting in significant single predictor (main or interaction) effects.

Nevertheless, concerning your current discussion, in psychology it is at least very common to test both main and interaction effects and we have learnt in our methods classes that it is possible to obtain interaction effects, although there are no main effects. I did not know that this is still not known to other research fields and that in some other research fields, people only test interaction effects when main effects are significant. Very interesting, since we in psychology preceed differently in this issue.

Todd,

sorry if I might have offended you in some way. This was not intended. I did not say that your statement is stupid, but I said that the "convention looking at interactions only when main effects are there" is IMO a stupid convention. So: I find the convention stupid - not your comments! Sorry if this was not mad clear enough.

A model is a tool that simplifies cetrain aspects of the world around us so that we may get some idea about systematic correlations between different observations. Models do not have the aim to be "true", not even the aim to model some "truth". They just provide a means to identify (and quantify) reliable patterns, that are useful for purpose with which we built the models ("all models are wrong, but some are useful", I think this is from Cox)

There are two way of looking at models: either a model is used to "explain" data given a theory (a model) about the relationships between the (observed) variables, or a model - or better the modelling approach - is used to find out possible relationships that fit to the data. I would call the first way "confirmatory" (check how good data fits to a given model), the second way "exploratory" (find a model than reasonably well fits the data).

The first way requires an a-priory specified model anyway. The model is justified by our theory. If such a model *should* contain interactions, it has to do so, even if the data at hand does not provide much evidence that such interaction is relevant.

For second way there are no clear rules for what is right and wrong. The models should be as simple as possible, but over-simplifications can also be bad. There is much literature about model/variable selection, and there is to my knowledge no agreement. Ideally, variables (main effects or interactions) are kept in the model when there is some expert knowledge about their relevace (what is often not the case, though). However, for such an explanatory approach it would be counter-productive to ignore interactios in the first place, because (1) one might miss relevant and interesting interactions and (2) one might miss other relevat ad iteresting effects that become clear only when interactions are considered. These are the two reasons why I find this "general approach of ignoring interactions in the absence of main effects" stupid.

Models with a *large* number of variables are rarely useful, because they are almost as complicated as the data. In such models, many interactions, especially of higher orders, can be included, rendering the interpretation next to impossible. Further, there are rarely enough data available to be able to provide evidence for higher-order interactions. So often there is no statistical justification of analyzing higher-order interactions. Therefore, the complexity of the model should be limited by the amout of available data and by the interpretability. This should answer your question "but at what level of interaction do you start?"

Models that can't be interpreted may have a commercial benefit (made just for the purpose of correct predictions) but they don't have much scientific value. In science, models are made to produce ideas.