Dear Ghaith Altawalbeh ,

Great question!

Yes, you are free to report the main effects only!

As the great Dr. Oscar Kempthorne said once (1975), “The testing of main effects in the presence of interaction, without additional input, is an exercise in fatuity”

This can be interpreted in two ways:

1) Certainly, when there is an interaction, interpretation of the the main effects is dangerous, as they do not illustrate some important relationships that exist between levels of the fixed effects used in the interaction!

2) However, the "without additional input" is certainly an important part of this quote! See discussion below.

For example, let's use a more pressing situation we have right now: COVID vaccination. Let's say that the overall effectiveness of two vaccines, namely V1 and V2, was 90 and 80% (P < 0.000001).

However, let's say that these vaccines were tested in 5 independent trials, following the same approaches. The effectiveness was the same across ALL trials (90% in the 5 trials, P = 0.9).

BUT, there was an interaction (P < 0.01) between vaccine type and trial, where in some trials, V1 had effectiveness ranging from 100 to 70%, and V2 had the same effectives in all trials (80%).

So, although there is a significant interaction (in this case, only for V1 actually), wouldn't it be VERY informative to say that V1 has an overall effectiveness of 90%? Even if is some trials these were, let's say, 70%, 100%, 90%, 85%? Would it be "an exercise of fatuity" (Kempthorne, 1975) to say this?

Likewise, would it be informative to ONLY restrict your findings to the interaction and say something like "Well, V1 is only 100% effective if we do exactly like in trial 1" - but, all trials we, theoretically, the same... so, this interaction is not so informative, right?

FINALLY... when you have interactions with SO many levels like you have (3 x 3 x 14 = 126), I would certainly expect that, if only some of all of these possible comparisons are real (reals as in the null hypothesis on some comparisons being true!), the overall F-test for this interaction will be diluted by all other true non-significant comparisons.

The question becomes: why are you looking at specific pair-wise comparisons? Are you simply testing all of them and then you found significant ones and then you want to discuss them? Then, I would suggest you NOT to report these. However, if these comparison were actually of interest PRIOR to the study, i.e., you had planned contrasts to test predetermined specific hypotheses, then, I am in favor of these being reported - this opens more discussion, but I think that for now, we have had enought.

Let me know if this helps! THanks, Nick

Dear Nick VL Serão ,

Huge thanks to the detailed and fast answer! Would you justify the following two Assumptions:

Did I understand your answer well?

Ghaith Altawalbeh ,

My pleasure - but, if my answer(s) is(are) helpful, feel free to "recommend" them ;)

For #1, I am not so sure what you mean by "summed up (ALL) conditions (=levels)". But, for simplification, in a two-way ANOVA, using a similar scenario of the vaccination example able, if there were 10 types of vaccines, 5 trials, and their interaction in the model, for example, if the main effects of vaccine and trial were significant and their interaction not, yes, for sure, you can simply report the effect of vaccine! Even if the effect of trial was significant. Does this answer your question?

As for #2, this is harder for me to understand what you mean... But, using my example above (#1 above, not the original answer), you will have 50 levels (10 vaccines x 5 trials) for the interaction. Continuing to assume that the interaction was not significant, EVEN if some of the 1225 (combination of 50 choose 2 = 1225) pairwise comparisons are significant, then, no, you still do not need to report the significant ones! Specially if you did not have predetermined comparisons made prior to the study...

The overall F-test of the interaction serves as a "protective" test - with 10 vaccines and 5 trials, you have (10-1) x (5-1) = 36 orthogonal (i.e., independent) questions being asked, and your F-test across all these 36 questions does not provide enough evidence that a general interaction exists between the two independent variables! Please notice that you have only 36 independent tests, whereas you have 1225 pairwise comparisons!!!!!! 34 times more comparisons that you "could"... Does this answer your question #2?

Thanks, Nick